Recurrent Neural Networks

Single Headed Attention RNN

Introduced by Merity in Single Headed Attention RNN: Stop Thinking With Your Head

SHA-RNN, or Single Headed Attention RNN, is a recurrent neural network, and language model when combined with an embedding input and softmax classifier, based on a core LSTM component and a single-headed attention module. Other design choices include a Boom feedforward layer and the use of layer normalization. The guiding principles of the author were to ensure simplicity in the architecture and to keep computational costs bounded (the model was originally trained with a single GPU).

Source: Single Headed Attention RNN: Stop Thinking With Your Head

Papers


Paper Code Results Date Stars

Tasks


Task Papers Share
Language Modelling 1 100.00%

Categories