1 code implementation • 4 Apr 2023 • Nafis Sadeq, Byungkyu Kang, Prarit Lamba, Julian McAuley
In this work, we propose an approach for influencing MLM pretraining in a way that can improve language model performance on a variety of knowledge-intensive tasks.