SentenceMIM: A Latent Variable Language Model

18 Feb 2020Micha LivneKevin SwerskyDavid J. Fleet

We introduce sentenceMIM, a probabilistic auto-encoder for language modelling, trained with Mutual Information Machine (MIM) learning. Previous attempts to learn variational auto-encoders for language data have had mixed success, with empirical performance well below state-of-the-art auto-regressive models, a key barrier being the occurrence of posterior collapse with VAEs... (read more)

PDF Abstract

Results from the Paper


 SOTA for Question Answering on YahooCQA (using extra training data)

     Get a GitHub badge
TASK DATASET MODEL METRIC NAME METRIC VALUE GLOBAL RANK USES EXTRA
TRAINING DATA
RESULT LEADERBOARD
Question Answering YahooCQA sMIM (1024) + [email protected] 0.753 # 1
MRR 0.861 # 1
Question Answering YahooCQA sMIM (1024) [email protected] 0.683 # 2
MRR 0.818 # 2