Holographic Reduced Representations are a simple mechanism to represent an associative array of keyvalue pairs in a fixedsize vector. Each individual keyvalue pair is the same size as the entire associative array; the array is represented by the sum of the pairs. Concretely, consider a complex vector key $r = (a_{r}[1]e^{iφ_{r}[1]}, a_{r}[2]e^{iφ_{r}[2]}, \dots)$, which is the same size as the complex vector value x. The pair is "bound" together by elementwise complex multiplication, which multiplies the moduli and adds the phases of the elements:
$$ y = r \otimes x $$
$$ y = \left(a_{r}[1]a_{x}[1]e^{i(φ_{r}[1]+φ_{x}[1])}, a_{r}[2]a_{x}[2]e^{i(φ_{r}[2]+φ_{x}[2])}, \dots\right) $$
Given keys $r_{1}$, $r_{2}$, $r_{3}$ and input vectors $x_{1}$, $x_{2}$, $x_{3}$, the associative array is:
$$c = r_{1} \otimes x_{1} + r_{2} \otimes x_{2} + r_{3} \otimes x_{3} $$
where we call $c$ a memory trace. Define the key inverse:
$$ r^{1} = \left(a_{r}[1]^{−1}e^{−iφ_{r}[1]}, a_{r}[2]^{−1}e^{−iφ_{r}[2]}, \dots\right) $$
To retrieve the item associated with key $r_{k}$, we multiply the memory trace elementwise by the vector $r^{1}_{k}$. For example:
$$ r_{2}^{−1} \otimes c = r_{2}^{1} \otimes \left(r_{1} \otimes x_{1} + r_{2} \otimes x_{2} + r_{3} \otimes x_{3}\right) $$
$$ r_{2}^{−1} \otimes c = x_{2} + r^{1}_{2} \otimes \left(r_{1} \otimes x_{1} + r_{3} \otimes x3\right) $$
$$ r_{2}^{−1} \otimes c = x_{2} + noise $$
The product is exactly $x_{2}$ together with a noise term. If the phases of the elements of the key vector are randomly distributed, the noise term has zero mean.
Source: Associative LSTMs
PAPER  DATE 

Associative Long ShortTerm Memory

20160209 
COMPONENT  TYPE 


🤖 No Components Found  You can add them if they exist; e.g. Mask RCNN uses RoIAlign 