LM-Critic: Language Models for Unsupervised Grammatical Error Correction

14 Sep 2021  ·  Michihiro Yasunaga, Jure Leskovec, Percy Liang ·

Training a model for grammatical error correction (GEC) requires a set of labeled ungrammatical / grammatical sentence pairs, but manually annotating such pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework has demonstrated strong results on learning to repair a broken program without any labeled examples, but this relies on a perfect critic (e.g., a compiler) that returns whether an example is valid or not, which does not exist for the GEC task... In this work, we show how to leverage a pretrained language model (LM) in defining an LM-Critic, which judges a sentence to be grammatical if the LM assigns it a higher probability than its local perturbations. We apply this LM-Critic and BIFI along with a large set of unlabeled sentences to bootstrap realistic ungrammatical / grammatical pairs for training a corrector. We evaluate our approach on GEC datasets across multiple domains (CoNLL-2014, BEA-2019, GMEG-wiki and GMEG-yahoo) and show that it outperforms existing methods in both the unsupervised setting (+7.7 F0.5) and the supervised setting (+0.5 F0.5). read more

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Grammatical Error Correction BEA-2019 (test) LM-Critic F0.5 72.9 # 3
Grammatical Error Correction CoNLL-2014 Shared Task LM-Critic F0.5 65.8 # 2
Grammatical Error Correction GMEG-wiki LM-Critic (Unsupervised) F0.5 50.6 # 1
Grammatical Error Correction GMEG-yahoo LM-Critic (Unsupervised) F0.5 52.2 # 1

Methods


No methods listed for this paper. Add relevant methods here