Stolen Probability: A Structural Weakness of Neural Language Models

1 code implementation • ACL 2020 • David Demeter, Gregory Kimmel, Doug Downey

Neural Network Language Models (NNLMs) generate probability distributions by applying a softmax function to a distance metric formed by taking the dot product of a prediction vector with all word vectors in a high-dimensional embedding space.

Inductive Bias