Search Results for author:
Found 18 papers, 10 papers with code
Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models
Pretrained large language models have become indispensable for solving various natural language processing (NLP) tasks.
Evaluating Parameter Efficient Learning for Generation
For cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al., 2019) performs the best amongst all the PERMs studied here, and (b) it outperforms finetuning if the task dataset is below a certain size.
Context Generation Improves Open Domain Question Answering
Prior work on closed-book QA either directly finetunes or prompts a pretrained language model (LM) to leverage the stored knowledge.
Factuality Enhanced Language Models for Open-Ended Text Generation
In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation.
Multi-Stage Prompting for Knowledgeable Dialogue Generation
We propose a multi-stage prompting approach to generate knowledgeable responses from a single pretrained LM.
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models.
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a key ingredient to the success of the model.
Ranked #2 on
Language Modelling
on LAMBADA
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
In this paper, we show how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters.
End-to-End Training of Neural Retrievers for Open-Domain Question Answering
We also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models.
Local Knowledge Powered Conversational Agents
State-of-the-art conversational agents have advanced significantly in conjunction with the use of large transformer-based language models.
BioMegatron: Larger Biomedical Domain Language Model
There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books.
Ranked #1 on
Named Entity Recognition (NER)
on BC5CDR-disease
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
We showcase the controllability of our model by replacing the keywords used to generate stories and re-running the generation process.
Large Scale Multi-Actor Generative Dialog Modeling
This work introduces the Generative Conversation Control model, an augmented and fine-tuned GPT-2 language model that conditions on past reference conversations to probabilistically model multi-turn conversations in the actor's persona.
Training Question Answering Models From Synthetic Data
On the SQuAD1. 1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQuAD1. 1 training set questions alone.
DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems
Extracting actionable insight from complex unlabeled scientific data is an open challenge and key to unlocking data-driven discovery in science.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8. 3 billion parameter transformer language model similar to GPT-2 and a 3. 9 billion parameter model similar to BERT.
Ranked #2 on
Language Modelling
on WikiText-103
(using extra training data)
Coloring Big Graphs with AlphaGoZero
We show that recent innovations in deep reinforcement learning can effectively color very large graphs -- a well-known NP-hard problem with clear commercial applications.
Language Modeling at Scale
We show how Zipf's Law can be used to scale up language modeling (LM) to take advantage of more training data and more GPUs.
