Mixtape: Breaking the Softmax Bottleneck Efficiently
The softmax bottleneck has been shown to limit the expressiveness of neural language models. Mixture of Softmaxes (MoS) is an effective approach to address such a theoretical limitation, but are expensive compared to softmax in terms of both memory and time. We propose Mixtape, an output layer that breaks the softmax bottleneck more efficiently with three novel techniques---logit space vector gating, sigmoid tree decomposition, and gate sharing. On four benchmarks including language modeling and machine translation, the Mixtape layer substantially improves the efficiency over the MoS layer by 3.5x to 10.5x while obtaining similar or better performance. A network equipped with Mixtape is only 20% to 34% slower than a softmax-based network with 10-30K vocabulary sizes, and outperforms softmax by a large margin in perplexity and translation quality. Notably, Mixtape achieves state-of-the-art results of 29.8 BLEU on WMT'14 English-German and 43.9 BLEU on WMT'14 English-French.
PDF Abstract
