Search Results for author:
Found 27 papers, 14 papers with code
Adding Instructions during Pretraining: Effective Way of Controlling Toxicity in Language Models
Pretrained large language models have become indispensable for solving various natural language processing (NLP) tasks.
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning
Augmenting pretrained language models (LMs) with a vision encoder (e. g., Flamingo) has obtained state-of-the-art results in image-to-text generation.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Evaluating Parameter Efficient Learning for Generation
For cross-domain and cross-dataset cases, we show that (a) Adapter (Houlsby et al., 2019) performs the best amongst all the PERMs studied here, and (b) it outperforms finetuning if the task dataset is below a certain size.
Context Generation Improves Open Domain Question Answering
Prior work on closed-book QA either directly finetunes or prompts a pretrained language model (LM) to leverage the stored knowledge.
Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models
We explore the idea of compressing the prompts used to condition language models, and show that compressed prompts can retain a substantive amount of information about the original prompt.
FP8 Formats for Deep Learning
FP8 is a natural progression for accelerating deep learning training inference beyond the 16-bit formats common in modern processors.
Factuality Enhanced Language Models for Open-Ended Text Generation
In this work, we measure and improve the factual accuracy of large-scale LMs for open-ended text generation.
Reducing Activation Recomputation in Large Transformer Models
In this paper, we show how to significantly accelerate training of large transformer models by reducing activation recomputation.
Multi-Stage Prompting for Knowledgeable Dialogue Generation
We propose a multi-stage prompting approach to generate knowledgeable responses from a single pretrained LM.
Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models.
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
Next, we detail the training process, the design of our training corpus, and our data curation techniques, which we believe is a key ingredient to the success of the model.
Ranked #2 on
Language Modelling
on LAMBADA
Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biases
We then provide the LM with instruction that consists of this subset of labeled exemplars, the query text to be classified, a definition of bias, and prompt it to make a decision.
Long-Short Transformer: Efficient Transformers for Language and Vision
For instance, Transformer-LS achieves 0. 97 test BPC on enwik8 using half the number of parameters than previous method, while being faster and is able to handle 3x as long sequences compared to its full-attention version on the same hardware.
Ranked #1 on
Language Modelling
on enwik8 dev
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
In this paper, we show how different types of parallelism methods (tensor, pipeline, and data parallelism) can be composed to scale to thousands of GPUs and models with trillions of parameters.
End-to-End Training of Neural Retrievers for Open-Domain Question Answering
We also explore two approaches for end-to-end supervised training of the reader and retriever components in OpenQA models.
Local Knowledge Powered Conversational Agents
State-of-the-art conversational agents have advanced significantly in conjunction with the use of large transformer-based language models.
BioMegatron: Larger Biomedical Domain Language Model
There has been an influx of biomedical domain-specific language models, showing language models pre-trained on biomedical text perform better on biomedical domain benchmarks than those trained on general domain text corpora such as Wikipedia and Books.
Ranked #1 on
Named Entity Recognition (NER)
on BC5CDR-disease
MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
We showcase the controllability of our model by replacing the keywords used to generate stories and re-running the generation process.
Large Scale Multi-Actor Generative Dialog Modeling
This work introduces the Generative Conversation Control model, an augmented and fine-tuned GPT-2 language model that conditions on past reference conversations to probabilistically model multi-turn conversations in the actor's persona.
Style Example-Guided Text Generation using Generative Adversarial Transformers
The style encoder extracts a style code from the reference example, and the text decoder generates texts based on the style code and the context.
Training Question Answering Models From Synthetic Data
On the SQuAD1. 1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQuAD1. 1 training set questions alone.
Neural ODEs for Image Segmentation with Level Sets
We propose a novel approach for image segmentation that combines Neural Ordinary Differential Equations (NODEs) and the Level Set method.
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
To demonstrate that large language models can further advance the state of the art (SOTA), we train an 8. 3 billion parameter transformer language model similar to GPT-2 and a 3. 9 billion parameter model similar to BERT.
Ranked #2 on
Language Modelling
on WikiText-103
(using extra training data)
Unsupervised Video Interpolation Using Cycle Consistency
We further introduce a pseudo supervised loss term that enforces the interpolated frames to be consistent with predictions of a pre-trained interpolation model.
Ranked #1 on
Video Frame Interpolation
on UCF101
(PSNR (sRGB) metric)
Trace norm regularization and faster inference for embedded speech recognition RNNs
We propose and evaluate new techniques for compressing and speeding up dense matrix multiplications as found in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR).
Deep Voice: Real-time Neural Text-to-Speech
We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural networks.
