Search Results for author:
Found 16 papers, 11 papers with code
Low Resource Text Classification with ULMFit and Backtranslation
A ULMFit model pretrained on wikitext103 and then fine-tuned on only 50 IMDB examples and 500 synthetic examples generated by backtranslation achieves 80. 6% accuracy, an 8. 1% improvement over the augmentation-free baseline with only 9 minutes of additional training time.
Using Small Proxy Datasets to Accelerate Hyperparameter Search
These "easy" proxies are higher quality than training on the full dataset for a reduced number of epochs (but equivalent computational cost), and, unexpectedly, higher quality than proxies constructed from the hardest examples.
Classification As Decoder: Trading Flexibility For Control In Neural Dialogue
Only 12% of our discriminative approach's responses are worse than the doctor's response in the same conversational context, compared to 18% for the generative model.
HuggingFace's Transformers: State-of-the-art Natural Language Processing
Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks.
Classification as Decoder: Trading Flexibility for Control in Medical Dialogue
Generative seq2seq dialogue systems are trained to predict the next word in dialogues that have already occurred.
Incrementally Improving Graph WaveNet Performance on Traffic Prediction
We present a series of modifications which improve upon Graph WaveNet's previously state-of-the-art performance on the METR-LA traffic prediction task.
Ranked #11 on
Traffic Prediction
on METR-LA
Transformers: State-of-the-Art Natural Language Processing
Transformer architectures have facilitated building higher-capacity models and pretraining has made it possible to effectively utilize this capacity for a wide variety of tasks.
Pre-trained Summarization Distillation
A third, simpler approach is to 'shrink and fine-tune' (SFT), which avoids any explicit distillation by copying parameters to a smaller student model and then fine-tuning.
8-bit Optimizers via Block-wise Quantization
To maintain stability and performance, we combine block-wise quantization with two additional changes: (1) dynamic quantization, a form of non-linear optimization that is precise for both large and small magnitude values, and (2) a stable embedding layer to reduce gradient variance that comes from the highly non-uniform distribution of input tokens in language models.
NormFormer: Improved Transformer Pretraining with Extra Normalization
The extra operations incur negligible compute cost (+0. 4% parameter increase), but improve pretraining perplexity and downstream task performance for both causal and masked language models ranging from 125 Million to 2. 7 Billion parameters.
Efficient Large Scale Language Modeling with Mixtures of Experts
This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning.
Few-shot Learning with Multilingual Language Models
Large-scale generative language models such as GPT-3 are competitive few-shot learners.
Efficient Language Modeling with Sparse all-MLP
All-MLP architectures have attracted increasing interest as an alternative to attention-based models.
Ranked #17 on
Question Answering
on StoryCloze
OPT: Open Pre-trained Transformer Language Models
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning.
Ranked #2 on
Stereotypical Bias Analysis
on CrowS-Pairs
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
It is widely acknowledged that large models have the potential to deliver superior performance across a broad range of domains.
