Search Results for author: Jingfei Du

Found 13 papers, 7 papers with code

On the Role of Bidirectionality in Language Model Pre-Training

no code implementations24 May 2022 Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, Ves Stoyanov

Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult.

Language Modelling Text Infilling

Few-shot Learning with Multilingual Language Models

1 code implementation20 Dec 2021 Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, Xian Li

In this work, we train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks.

Few-Shot Learning Hate Speech Detection +4

Larger-Scale Transformers for Multilingual Masked Language Modeling

no code implementations ACL (RepL4NLP) 2021 Naman Goyal, Jingfei Du, Myle Ott, Giri Anantharaman, Alexis Conneau

Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0. 3% on average while handling 99 more languages.

Language Modelling Masked Language Modeling

Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning

1 code implementation ICLR 2021 Beliz Gunel, Jingfei Du, Alexis Conneau, Ves Stoyanov

Our proposed fine-tuning objective leads to models that are more robust to different levels of noise in the fine-tuning training data, and can generalize better to related tasks with limited labeled data.

Contrastive Learning Data Augmentation +3

Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval

1 code implementation ICLR 2021 Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Wen-tau Yih, Sebastian Riedel, Douwe Kiela, Barlas Oğuz

We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datasets, HotpotQA and multi-evidence FEVER.

Question Answering

General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference

no code implementations Findings of the Association for Computational Linguistics 2020 Jingfei Du, Myle Ott, Haoran Li, Xing Zhou, Veselin Stoyanov

The resulting method offers a compelling solution for using large-scale pre-trained models at a fraction of the computational cost when multiple tasks are performed on the same text.

Knowledge Distillation Quantization

Cannot find the paper you are looking for? You can Submit a new open access paper.