Search Results for author:
Found 13 papers, 7 papers with code
On the Role of Bidirectionality in Language Model Pre-Training
Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult.
Improving In-Context Few-Shot Learning via Self-Supervised Training
Self-supervised pretraining has made few-shot learning possible for many NLP tasks.
Few-shot Learning with Multilingual Language Models
In this work, we train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages, and study their few- and zero-shot learning capabilities in a wide range of tasks.
Efficient Large Scale Language Modeling with Mixtures of Experts
Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation.
Larger-Scale Transformers for Multilingual Masked Language Modeling
Our model also outperforms the RoBERTa-Large model on several English tasks of the GLUE benchmark by 0. 3% on average while handling 99 more languages.
Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
Our proposed fine-tuning objective leads to models that are more robust to different levels of noise in the fine-tuning training data, and can generalize better to related tasks with limited labeled data.
Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art
A large array of pretrained models are available to the biomedical NLP (BioNLP) community.
Self-training Improves Pre-training for Natural Language Understanding
Unsupervised pre-training has led to much recent progress in natural language understanding.
Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
We propose a simple and efficient multi-hop dense retrieval approach for answering complex open-domain questions, which achieves state-of-the-art performance on two multi-hop datasets, HotpotQA and multi-evidence FEVER.
Ranked #14 on
Question Answering
on HotpotQA
General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
The resulting method offers a compelling solution for using large-scale pre-trained models at a fraction of the computational cost when multiple tasks are performed on the same text.
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Models trained with our new objective yield significant improvements on the fact completion task.
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.
Ranked #2 on
Common Sense Reasoning
on SWAG
Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition
Our Knowledge-Augmented Language Model (KALM) continues this line of work by augmenting a traditional model with a KB.
