Holographic Reduced Representation

Holographic Reduced Representations are a simple mechanism to represent an associative array of key-value pairs in a fixed-size vector. Each individual key-value pair is the same size as the entire associative array; the array is represented by the sum of the pairs. Concretely, consider a complex vector key $r = (a_{r}[1]e^{iφ_{r}[1]}, a_{r}[2]e^{iφ_{r}[2]}, \dots)$, which is the same size as the complex vector value x. The pair is "bound" together by element-wise complex multiplication, which multiplies the moduli and adds the phases of the elements:

$$ y = r \otimes x $$

$$ y = \left(a_{r}[1]a_{x}[1]e^{i(φ_{r}[1]+φ_{x}[1])}, a_{r}[2]a_{x}[2]e^{i(φ_{r}[2]+φ_{x}[2])}, \dots\right) $$

Given keys $r_{1}$, $r_{2}$, $r_{3}$ and input vectors $x_{1}$, $x_{2}$, $x_{3}$, the associative array is:

$$c = r_{1} \otimes x_{1} + r_{2} \otimes x_{2} + r_{3} \otimes x_{3} $$

where we call $c$ a memory trace. Define the key inverse:

$$ r^{-1} = \left(a_{r}[1]^{−1}e^{−iφ_{r}[1]}, a_{r}[2]^{−1}e^{−iφ_{r}[2]}, \dots\right) $$

To retrieve the item associated with key $r_{k}$, we multiply the memory trace element-wise by the vector $r^{-1}_{k}$. For example:

$$ r_{2}^{−1} \otimes c = r_{2}^{-1} \otimes \left(r_{1} \otimes x_{1} + r_{2} \otimes x_{2} + r_{3} \otimes x_{3}\right) $$

$$ r_{2}^{−1} \otimes c = x_{2} + r^{-1}_{2} \otimes \left(r_{1} \otimes x_{1} + r_{3} \otimes x3\right) $$

$$ r_{2}^{−1} \otimes c = x_{2} + noise $$

The product is exactly $x_{2}$ together with a noise term. If the phases of the elements of the key vector are randomly distributed, the noise term has zero mean.

Source: Associative LSTMs