Search Results for author:
Found 36 papers, 19 papers with code
MultiReQA: A Cross-Domain Evaluation forRetrieval Question Answering Models
Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al., 2019). This dataset paper presents MultiReQA, a new multi-domain ReQA evaluation suite composed of eight retrieval QA tasks drawn from publicly available QA datasets.
Training LLMs over Neurally Compressed Text
In this paper, we explore the idea of training large language models (LLMs) over highly compressed text.
Transfer Learning for Text Diffusion Models
In this report, we explore the potential for text diffusion to replace autoregressive (AR) decoding for the training and deployment of large language models (LLMs).
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models
To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times.
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment.
FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation
Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked.
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining
As part of our contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.
Character-Aware Models Improve Visual Text Rendering
In the text-only domain, we find that character-aware models provide large gains on a novel spelling task (WikiSpell).
FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation
We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation, a type of style-targeted translation.
Bidirectional Language Models Are Also Few-shot Learners
An arbitrary task can be reformulated as a natural language prompt, and a language model can be asked to generate the completion, indirectly performing the task in a paradigm known as prompt-based learning.
Reducing Retraining by Recycling Parameter-Efficient Prompts
Parameter-efficient methods are able to use a single frozen pre-trained large language model (LLM) to perform many tasks by learning task-specific soft prompts that modulate model behavior when concatenated to the input text.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation
In this paper, we explore the challenging problem of performing a generative task in a target language when labeled data is only available in English, using summarization as a case study.
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
Finally, we propose an efficient retrieval approach that interprets task prompts as task embeddings to identify similar tasks and predict the most transferable source tasks for a novel target task.
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
To support our investigation, we establish a new sentence representation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark.
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models?
Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks.
Towards Universality in Multilingual Text Rewriting
In this work, we take the first steps towards building a universal rewriter: a model capable of rewriting text in any language to exhibit a wide variety of attributes, including styles and languages, while preserving as much of the original semantics as possible.
nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?
Recently, mT5 - a massively multilingual version of T5 - leveraged a unified text-to-text format to attain state-of-the-art results on a wide variety of multilingual NLP tasks.
ByT5: Towards a token-free future with pre-trained byte-to-byte models
Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units.
Ranked #1 on
Cross-Lingual Natural Language Inference
on XNLI
Cross-Lingual Natural Language Inference
Cross-Lingual NER
+3
The Power of Scale for Parameter-Efficient Prompt Tuning
More remarkably, through ablations on model size using T5, we show that prompt tuning becomes more competitive with scale: as models exceed billions of parameters, our method "closes the gap" and matches the strong performance of model tuning (where all model weights are tuned).
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation
While a sizeable gap to human-level performance remains, improvements have been easier to achieve in some tasks than in others.
Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution
We propose a straightforward vocabulary adaptation scheme to extend the language capacity of multilingual machine translation models, paving the way towards efficient continual learning for multilingual machine translation.
Towards Zero-Shot Multilingual Synthetic Question and Answer Generation for Cross-Lingual Reading Comprehension
We propose a simple method to generate multilingual question and answer pairs on a large scale through the use of a single generative model.
mT5: A massively multilingual pre-trained text-to-text transformer
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks.
Ranked #2 on
Reading Comprehension
on MuSeRC
TextSETTR: Few-Shot Text Style Extraction and Tunable Targeted Restyling
Unlike previous approaches requiring style-labeled training data, our method makes use of readily-available unlabeled text by relying on the implicit connection in style between adjacent sentences, and uses labeled data only at inference time.
TextSETTR: Label-Free Text Style Extraction and Tunable Targeted Restyling
We present a novel approach to the challenging problem of label-free text style transfer.
MultiReQA: A Cross-Domain Evaluation for Retrieval Question Answering Models
Retrieval question answering (ReQA) is the task of retrieving a sentence-level answer to a question from an open corpus (Ahmad et al., 2019). This paper presents MultiReQA, anew multi-domain ReQA evaluation suite com-posed of eight retrieval QA tasks drawn from publicly available QA datasets.
LAReQA: Language-agnostic answer retrieval from a multilingual pool
We present LAReQA, a challenging new benchmark for language-agnostic answer retrieval from a multilingual candidate pool.
Bridging the Gap for Tokenizer-Free Language Models
Purely character-based language models (LMs) have been lagging in quality on large scale datasets, and current state-of-the-art LMs rely on word tokenization.
ReQA: An Evaluation for End-to-End Answer Retrieval Models
Popular QA benchmarks like SQuAD have driven progress on the task of identifying answer spans within a specific passage, with models now surpassing human performance.
Multilingual Universal Sentence Encoder for Semantic Retrieval
We introduce two pre-trained retrieval focused multilingual sentence encoding models, respectively based on the Transformer and CNN model architectures.
Universal Sentence Encoder for English
We present easy-to-use TensorFlow Hub sentence embedding models having good task transfer performance.
Character-Level Language Modeling with Deeper Self-Attention
LSTMs and other RNN variants have shown strong performance on character-level language modeling.
Ranked #8 on
Language Modelling
on Hutter Prize
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings
This paper presents an effective approach for parallel corpus mining using bilingual sentence embeddings.
Learning Semantic Textual Similarity from Conversations
We present a novel approach to learn representations for sentence-level semantic similarity using conversational data.
Universal Sentence Encoder
For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance.
Ranked #2 on
Text Classification
on TREC-6
Conversational Response Selection
Semantic Textual Similarity
+7
