Search Results for author:
Found 77 papers, 53 papers with code
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP).
Ranked #1 on
Sentiment Analysis
on SST-2 Binary classification
mT5: A massively multilingual pre-trained text-to-text transformer
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks.
Ranked #2 on
Reading Comprehension
on MuSeRC
ByT5: Towards a token-free future with pre-trained byte-to-byte models
Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.
Ranked #1 on
Cross-Lingual Natural Language Inference
on XNLI
Cross-Lingual Natural Language Inference
Cross-Lingual NER
+3
How Much Knowledge Can You Pack Into the Parameters of a Language Model?
It has recently been observed that neural language models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries.
Do Transformer Modifications Transfer Across Implementations and Applications?
The research community has proposed copious modifications to the Transformer architecture since it was introduced over three years ago, relatively few of which have seen widespread adoption.
Learning a Latent Space of Multitrack Measures
Discovering and exploring the underlying structure of multi-instrumental music using learning-based approaches remains an open problem.
Theano: A Python framework for fast computation of mathematical expressions
Since its introduction, it has been one of the most used CPU and GPU mathematical compilers - especially in the machine learning community - and has shown steady performance improvements.
Petals: Collaborative Inference and Fine-tuning of Large Models
However, these techniques have innate limitations: offloading is too slow for interactive inference, while APIs are not flexible enough for research that requires access to weights, attention or logits.
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Multitask Prompted Training Enables Zero-Shot Task Generalization
Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020).
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
PromptSource is a system for creating, sharing, and using natural language prompts.
Scaling Up Models and Data with $\texttt{t5x}$ and $\texttt{seqio}$
Recent neural network-based language models have benefited greatly from scaling up the size of training datasets and the number of parameters in the models themselves.
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
ICL incurs substantial computational, memory, and storage costs because it involves processing all of the training examples every time a prediction is made.
Ranked #1 on
Few-Shot Text Classification
on RAFT
What Language Model to Train if You Have One Million GPU Hours?
All our models and code are open-sourced at https://huggingface. co/bigscience .
MixMatch: A Holistic Approach to Semi-Supervised Learning
Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets.
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring
Distribution alignment encourages the marginal distribution of predictions on unlabeled data to be close to the marginal distribution of ground-truth labels.
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance.
Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems
We propose a simplified model of attention which is applicable to feed-forward neural networks and demonstrate that the resulting model can solve the synthetic "addition" and "multiplication" long-term memory problems for sequence lengths which are both longer and more widely varying than the best published results for these tasks.
DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows
The rapid rise to prominence of these models and these unique challenges has had immediate adverse impacts on open science and on the reproducibility of work that uses them.
Crosslingual Generalization through Multitask Finetuning
We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus.
Ranked #1 on
Question Answering
on StoryCloze
Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
However, we argue that these benchmarks fail to address many issues that these algorithms would face in real-world applications.
Scaling Data-Constrained Language Models
We find that with constrained data for a fixed compute budget, training with up to 4 epochs of repeated data yields negligible changes to loss compared to having unique data.
Deduplicating Training Data Mitigates Privacy Risks in Language Models
Past work has shown that large language models are susceptible to privacy attacks, where adversaries generate sequences from a trained model and detect which sequences are memorized from the training set.
Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer
Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code.
Onsets and Frames: Dual-Objective Piano Transcription
We advance the state of the art in polyphonic piano music transcription by using a deep convolutional and recurrent neural network which is trained to jointly predict onsets and frames.
Learning to Route Among Specialized Experts for Zero-Shot Generalization
Unlike past methods that learn to route among specialized models, PHATGOOSE explores the possibility that zero-shot generalization will be improved if different experts can be adaptively chosen for each token and at each layer in the model.
Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models
Currently, most machine learning models are trained by centralized teams and are rarely updated.
Extracting Training Data from Large Language Models
We demonstrate our attack on GPT-2, a language model trained on scrapes of the public Internet, and are able to extract hundreds of verbatim text sequences from the model's training data.
Improving and Simplifying Pattern Exploiting Training
However, PET uses task-specific unlabeled data.
ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring
We improve the recently-proposed ``MixMatch semi-supervised learning algorithm by introducing two new techniques: distribution alignment and augmentation anchoring.
TIES-Merging: Resolving Interference When Merging Models
To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign.
Online and Linear-Time Attention by Enforcing Monotonic Alignments
Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems.
Ranked #20 on
Speech Recognition
on TIMIT
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?
In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder-only and encoder-decoder), trained with two different pretraining objectives (autoregressive and masked language modeling), and evaluated with and without multitask prompted finetuning.
Large Language Models Struggle to Learn Long-Tail Knowledge
The Internet contains a wealth of knowledge -- from the birthdays of historical figures to tutorials on how to code -- all of which may be learned by language models.
A Survey on Data Selection for Language Models
A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training.
Robust and Generalizable Visual Representation Learning via Random Convolutions
In this work, we show that the robustness of neural networks can be greatly improved through the use of random convolutions as data augmentation.
Ranked #110 on
Domain Generalization
on PACS
Training Neural Networks with Fixed Sparse Masks
In this paper, we show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations.
Monotonic Chunkwise Attention
Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction.
A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music
The Variational Autoencoder (VAE) has proven to be an effective model for producing semantically meaningful latent representations for natural data.
Merging Models with Fisher-Weighted Averaging
Computing a simple average of the models' parameters therefore corresponds to making an isotropic Gaussian approximation to their posteriors.
Top-k Training of GANs: Improving GAN Performance by Throwing Away Bad Samples
We introduce a simple (one line of code) modification to the Generative Adversarial Network (GAN) training algorithm that materially improves results with no increase in computational cost: When updating the generator parameters, we simply zero out the gradient contributions from the elements of the batch that the critic scores as `least realistic'.
Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
While large language models have proven effective in a huge range of downstream applications, they often generate text that is problematic or lacks a desired attribute.
Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition
Adversarial examples are inputs to machine learning models designed by an adversary to cause an incorrect output.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+2
Improving Few-Shot Generalization by Exploring and Exploiting Auxiliary Data
In this work, we focus on Few-shot Learning with Auxiliary Data (FLAD), a training paradigm that assumes access to auxiliary data during few-shot learning in hopes of improving generalization.
WT5?! Training Text-to-Text Models to Explain their Predictions
Neural networks have recently achieved human-level performance on various challenging natural language processing (NLP) tasks, but it is notoriously difficult to understand why a neural network produced a particular prediction.
Evaluating the Factual Consistency of Large Language Models Through News Summarization
To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent.
ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
Despite the efficiency of PEFT methods, the size of expert models can make it onerous to retrieve expert models per query over high-latency networks like the Internet or serve multiple experts on a single GPU.
Merging by Matching Models in Task Parameter Subspaces
Model merging aims to cheaply combine individual task-specific models into a single multitask model.
Training a Subsampling Mechanism in Expectation
We describe a mechanism for subsampling sequences and show how to compute its expected output so that it can be trained with standard backpropagation.
NPEFF: Non-Negative Per-Example Fisher Factorization
Using unique properties of NPEFF's parameter-space representations, we ran extensive experiments to verify that the connections between directions in parameters space and examples recovered by NPEFF actually reflect the model's processing.
Is Generator Conditioning Causally Related to GAN Performance?
Motivated by this, we study the distribution of singular values of the Jacobian of the generator in Generative Adversarial Networks (GANs).
Learning Hard Alignments with Variational Inference
There has recently been significant interest in hard attention models for tasks such as object recognition, visual captioning and speech recognition.
Poker-CNN: A Pattern Learning Strategy for Making Draws and Bets in Poker Games
The contributions of this paper include: (1) a novel representation for poker games, extendable to different poker variations, (2) a CNN based learning model that can effectively learn the patterns in three different games, and (3) a self-trained system that significantly beats the heuristic-based program on which it is trained, and our system is competitive against human expert players.
Thermometer Encoding: One Hot Way To Resist Adversarial Examples
It is well known that it is possible to construct "adversarial examples" for neural networks: inputs which are misclassified by the network yet indistinguishable from true data.
Monotonic Infinite Lookback Attention for Simultaneous Machine Translation
Simultaneous machine translation begins to translate each source sentence before the source speaker is finished speaking, with applications to live and streaming scenarios.
Detecting and Diagnosing Adversarial Images with Class-Conditional Capsule Reconstructions
Then, we diagnose the adversarial examples for CapsNets and find that the success of the reconstructive attack is highly related to the visual similarity between the source and target class.
Towards GAN Benchmarks Which Require Generalization
For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic.
Deflecting Adversarial Attacks
There has been an ongoing cycle where stronger defenses against adversarial attacks are subsequently broken by a more advanced defense-aware attack.
A Large-scale Study on Training Sample Memorization in Generative Modeling
Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric.
NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
We review the EfficientQA competition from NeurIPS 2020.
On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
Many recent developments on generative models for natural images have relied on heuristically-motivated metrics that can be easily gamed by memorizing a small sample from the true distribution or training a model directly to improve the metric.
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP
NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets.
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
What are the units of text that we want to model?
Learning with Limited Text Data
Natural Language Processing (NLP) has achieved great progress in the past decade on the basis of neural models, which often make use of large amounts of labeled data to achieve state-of-the-art performance.
Emergent Abilities of Large Language Models
Scaling up language models has been shown to predictably improve performance and sample efficiency on a wide range of downstream tasks.
Efficient Methods for Natural Language Processing: A Survey
Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows.
Bidirectional Language Models Are Also Few-shot Learners
An arbitrary task can be reformulated as a natural language prompt, and a language model can be asked to generate the completion, indirectly performing the task in a paradigm known as prompt-based learning.
Compositional Generalization in Unsupervised Compositional Representation Learning: A Study on Disentanglement and Emergent Language
In hopes of enabling compositional generalization, various unsupervised learning algorithms have been proposed with inductive biases that aim to induce compositional structure in learned representations (e. g. disentangled representation and emergent language learning).
A Combinatorial Perspective on the Optimization of Shallow ReLU Networks
We explore the implications of this combinatorial aspect of ReLU optimization in this work.
ColD Fusion: Collaborative Descent for Distributed Multitask Finetuning
We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion.
Knowledge is a Region in Weight Space for Fine-tuned Language Models
Notably, we show that language models that have been finetuned on the same dataset form a tight cluster in the weight space, while models finetuned on different datasets from the same underlying task form a looser cluster.
Soft Merging of Experts with Adaptive Routing
To address this issue, we introduce Soft Merging of Experts with Adaptive Routing (SMEAR), which avoids discrete routing by using a single "merged" expert constructed via a weighted average of all of the experts' parameters.
Efficient Online Data Mixing For Language Model Pre-Training
The data used to pretrain large language models has a decisive impact on a model's downstream performance, which has led to a large body of work on data selection methods that aim to automatically determine the most suitable data to use for pretraining.
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters.
Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
Mixture-of-Experts (MoE) language models can reduce computational costs by 2-4$\times$ compared to dense models without sacrificing performance, making them more efficient in computation-bounded scenarios.
