Search Results for author:
Found 81 papers, 47 papers with code
Branch-Solve-Merge Improves Large Language Model Evaluation and Generation
Large Language Models (LLMs) are frequently used for multi-faceted language generation and evaluation tasks that involve satisfying intricate user constraints or taking into account multiple aspects and criteria.
The Temporal Structure of Language Processing in the Human Brain Corresponds to The Layered Hierarchy of Deep Language Models
Our results reveal a connection between human language processing and DLMs, with the DLM's layer-by-layer accumulation of contextual information mirroring the timing of neural activity in high-order language areas.
Self-Alignment with Instruction Backtranslation
We present a scalable method to build a high quality instruction following language model by automatically labelling human-written text with corresponding instructions.
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding
We introduce ZeroSCROLLS, a zero-shot benchmark for natural language understanding over long texts, which contains only test and small validation sets, without training data.
LIMA: Less Is More for Alignment
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences.
Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
Using this web app we build Pick-a-Pic, a large, open dataset of text-to-image prompts and real users' preferences over generated images.
Vision Transformers with Mixed-Resolution Tokenization
In this work, we apply this approach to Vision Transformers by introducing a novel image tokenization scheme, replacing the standard uniform grid with a mixed-resolution sequence of tokens, where each token represents a patch of arbitrary size.
X&Fuse: Fusing Visual Information in Text-to-Image Generation
We introduce X&Fuse, a general approach for conditioning on visual information when generating images from text.
Scaling Laws for Generative Mixed-Modal Language Models
To better understand the scaling properties of such mixed-modal models, we conducted over 250 experiments using seven different modalities and model sizes ranging from 8 million to 30 billion, trained on 5-100 billion tokens.
Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
We collect 64, 000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth.
A Simple Baseline for Beam Search Reranking
Reranking methods in machine translation aim to close the gap between common evaluation metrics (e. g. BLEU) and maximum likelihood learning and decoding algorithms.
Causes and Cures for Interference in Multilingual Translation
Multilingual machine translation models can benefit from synergy between different language pairs, but also suffer from interference.
LMentry: A Language Model Benchmark of Elementary Language Tasks
As the performance of large language models rapidly improves, benchmarks are getting larger and more complex as well.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Large language models are able to perform a task by conditioning on a few input-output demonstrations - a paradigm known as in-context learning.
Breaking Character: Are Subwords Good Enough for MRLs After All?
Large pretrained language models (PLMs) typically tokenize the input string into contiguous subwords before any pretraining or inference.
Transformer Language Models without Positional Encodings Still Learn Positional Information
Causal transformer language models (LMs), such as GPT-3, typically require some form of positional encoding, such as positional embeddings.
Are Mutually Intelligible Languages Easier to Translate?
Two languages are considered mutually intelligible if their native speakers can communicate with each other, while using their own mother tongue.
SCROLLS: Standardized CompaRison Over Long Language Sequences
NLP benchmarks have largely focused on short texts, such as sentences and paragraphs, even though long texts comprise a considerable amount of natural language in the wild.
Ranked #8 on
Long-range modeling
on SCROLLS
Simple Local Attentions Remain Competitive for Long-Context Tasks
Many NLP tasks require processing long contexts beyond the length limit of pretrained models.
Learning to Retrieve Passages without Supervision
Dense retrievers for open-domain question answering (ODQA) have been shown to achieve impressive performance by training on large datasets of question-passage pairs.
A Few More Examples May Be Worth Billions of Parameters
We investigate the dynamics of increasing the number of model parameters versus the number of labeled examples across a wide variety of tasks.
ParaShoot: A Hebrew Question Answering Dataset
NLP research in Hebrew has largely focused on morphology and syntax, where rich annotated datasets in the spirit of Universal Dependencies are available.
Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens
Standard pretrained language models operate on sequences of subword tokens without direct access to the characters that compose each token's string representation.
How Optimal is Greedy Decoding for Extractive Question Answering?
However, this approach does not ensure that the answer is a span in the given passage, nor does it guarantee that it is the most probable one.
What Do You Get When You Cross Beam Search with Nucleus Sampling?
We combine beam search with the probabilistic pruning technique of nucleus sampling to create two deterministic nucleus search algorithms for natural language generation.
Can Latent Alignments Improve Autoregressive Machine Translation?
Latent alignment objectives such as CTC and AXE significantly improve non-autoregressive machine translation models.
How to Train BERT with an Academic Budget
While large language models a la BERT are used ubiquitously in NLP, pretraining them is considered a luxury that only a few well-funded industry labs can afford.
Ranked #19 on
Question Answering
on Quora Question Pairs
Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language
Current NLP datasets targeting ambiguity can be solved by a native speaker with relative ease.
Coreference Resolution without Span Representations
The introduction of pretrained language models has reduced many complex task-specific NLP models to simple lightweight layers.
Ranked #5 on
Coreference Resolution
on CoNLL 2012
Few-Shot Question Answering by Pretraining Span Selection
Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the passage for each masked span.
Transformer Feed-Forward Layers Are Key-Value Memories
Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored.
The Turking Test: Can Language Models Understand Instructions?
Supervised machine learning provides the learner with a set of input-output examples of the target task.
Neural Machine Translation without Embeddings
Many NLP models operate over sequences of subword tokens produced by hand-crafted tokenization rules and heuristic subword induction algorithms.
Aligned Cross Entropy for Non-Autoregressive Machine Translation
This difficultly is compounded during training with cross entropy loss, which can highly penalize small shifts in word order.
Semi-Autoregressive Training Improves Mask-Predict Decoding
The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach.
Improving Transformer Models by Reordering their Sublayers
Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers.
Ranked #7 on
Language Modelling
on enwik8
Blockwise Self-Attention for Long Document Understanding
We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies.
Generalization through Memorization: Nearest Neighbor Language Models
Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15. 79 - a 2. 9 point improvement with no additional training.
Ranked #9 on
Language Modelling
on WikiText-103
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
We evaluate a number of noising approaches, finding the best performance by both randomly shuffling the order of the original sentences and using a novel in-filling scheme, where spans of text are replaced with a single mask token.
Ranked #3 on
Open-Domain Question Answering
on ELI5
Structural Language Models of Code
We introduce a new approach to any-code completion that leverages the strict syntax of programming languages to model a code snippet as a tree - structural language modeling (SLM).
Structural Language Models for Any-Code Generation
We introduce a new approach to AnyGen that leverages the strict syntax of programming languages to model a code snippet as tree structural language modeling (SLM).
BERT for Coreference Resolution: Baselines and Analysis
We apply BERT to coreference resolution, achieving strong improvements on the OntoNotes (+3. 9 F1) and GAP (+11. 5 F1) benchmarks.
Ranked #10 on
Coreference Resolution
on CoNLL 2012
(using extra training data)
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging.
Ranked #2 on
Text Classification
on arXiv-10
SpanBERT: Improving Pre-training by Representing and Predicting Spans
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text.
Ranked #1 on
Question Answering
on NewsQA
(F1 metric)
What Does BERT Look At? An Analysis of BERT's Attention
Large pre-trained neural networks such as BERT have had great recent success in NLP, motivating a growing body of research investigating what aspects of language they are able to learn from unlabeled data.
Are Sixteen Heads Really Better than One?
Attention is a powerful and ubiquitous mechanism for allowing neural models to focus on particular salient pieces of information by taking their weighted average when making predictions.
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks.
Mask-Predict: Parallel Decoding of Conditional Masked Language Models
Most machine translation systems generate text autoregressively from left to right.
Training on Synthetic Noise Improves Robustness to Natural Noise in Machine Translation
We consider the problem of making machine translation more robust to character-level variation at the source side, such as typos.
pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference
Reasoning about implied relationships (e. g., paraphrastic, common sense, encyclopedic) between pairs of words is crucial for many cross-sentence inference problems.
code2seq: Generating Sequences from Structured Representations of Code
The ability to generate natural language sequences from source code snippets has a variety of applications such as code summarization, documentation, and retrieval.
Ultra-Fine Entity Typing
We introduce a new entity typing task: given a sentence with an entity mention, the goal is to predict a set of free-form phrases (e. g. skyscraper, songwriter, or criminal) that describe appropriate types for the target entity.
Ranked #4 on
Entity Typing
on Ontonotes v5 (English)
LSTMs Exploit Linguistic Attributes of Data
While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data.
Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling
Recent BIO-tagging-based neural semantic role labeling models are very high performing, but assume gold predicates as part of the input and cannot incorporate span-level features.
Deep RNNs Encode Soft Hierarchical Syntax
We present a set of experiments to demonstrate that deep recurrent neural networks (RNNs) learn internal representations that capture soft hierarchical notions of syntax from highly varied supervision.
Long Short-Term Memory as a Dynamically Computed Element-wise Weighted Sum
LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections.
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset.
Ranked #36 on
Natural Language Inference
on MultiNLI
Natural Language Inference
Natural Language Understanding
+2
code2vec: Learning Distributed Representations of Code
We demonstrate the effectiveness of our approach by using it to predict a method's name from the vector representation of its body.
A General Path-Based Representation for Predicting Program Properties
A major challenge when learning from programs is $\textit{how to represent programs in a way that facilitates effective learning}$.
Annotation Artifacts in Natural Language Inference Data
Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to.
Simulating Action Dynamics with Neural Process Networks
Understanding procedural language requires anticipating the causal effects of actions, even when they are not explicitly stated.
Named Entity Disambiguation for Noisy Text
We address the task of Named Entity Disambiguation (NED) for noisy text.
Zero-Shot Relation Extraction via Reading Comprehension
We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot.
Recurrent Additive Networks
We introduce recurrent additive networks (RANs), a new gated RNN which is distinguished by the use of purely additive latent state updates.
Modeling Extractive Sentence Intersection via Subtree Entailment
Sentence intersection captures the semantic overlap of two texts, generalizing over paradigms such as textual entailment and semantic text similarity.
Abstractive Text Summarization
Natural Language Inference
+1
A Strong Baseline for Learning Cross-Lingual Word Embeddings from Sentence Alignments
While cross-lingual word embeddings have been studied extensively in recent years, the qualitative differences between the different algorithms remain vague.
Improving Distributional Similarity with Lessons Learned from Word Embeddings
Recent trends suggest that neural-network-inspired word embedding models outperform traditional count-based distributional models on word similarity and analogy detection tasks.
Neural Word Embedding as Implicit Matrix Factorization
We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant.
word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method
The word2vec software of Tomas Mikolov and colleagues (https://code. google. com/p/word2vec/ ) has gained a lot of traction lately, and provides state-of-the-art word embeddings.
