BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

WS 2019 · Alex Wang, Kyunghyun Cho ·

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

PDF Abstract WS 2019 PDF WS 2019 Abstract

Code

Add Remove Mark official

kyunghyuncho/bert-gen official

317

nyu-dl/bert-gen official

317

seanrjohnson/protein_gibbs_sampler

30

opqrstuvcut/BertMouth

11

gandharvsuri/Conditional-Generative…

2

See all 11 implementations

Tasks

Add Remove

Language Modelling

Datasets

Results from the Paper

Edit

Submit results from this paper to get state-of-the-art GitHub badges and help the community compare results to other papers.

Methods

Add Remove

Adam • Attention Dropout • BERT • Dense Connections • Dropout • GELU • Layer Normalization • Linear Layer • Linear Warmup With Linear Decay • Multi-Head Attention • Residual Connection • Scaled Dot-Product Attention • Softmax • Weight Decay • WordPiece