Search Results for author:
Found 95 papers, 40 papers with code
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks.
Ranked #1 on
Natural Language Inference
on QNLI
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
Gaussian Error Linear Units (GELUs)
We propose the Gaussian Error Linear Unit (GELU), a high-performing neural network activation function.
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
We consider the two related problems of detecting if an example is misclassified or out-of-distribution.
Ranked #12 on
Out-of-Distribution Detection
on CIFAR-10 vs CIFAR-100
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks
We propose syntactically controlled paraphrase networks (SCPNs) and use them to generate adversarial examples.
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise
We utilize trusted data by proposing a loss correction technique that utilizes trusted examples in a data-efficient manner to mitigate the effects of label noise on deep neural network classifiers.
Simple and Effective Paraphrastic Similarity from Parallel Translations
We present a model and methodology for learning paraphrastic sentence embeddings directly from bitext, removing the time-consuming intermediate step of creating paraphrase corpora.
Paraphrastic Representations at Scale
We train these models on large amounts of data, achieving significantly improved performance from the original papers proposing the methods on a suite of monolingual semantic similarity, cross-lingual semantic similarity, and bitext mining tasks.
A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations
We propose a generative model for a sentence that uses two latent variables, with one intended to represent the syntax of the sentence and the other to represent its semantics.
Beyond BLEU: Training Neural Machine Translation with Semantic Similarity
While most neural machine translation (NMT) systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can substantially improve final translation accuracy.
Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
Prior work on pretrained sentence embeddings and benchmarks focus on the capabilities of stand-alone sentences.
Variational Sequential Labelers for Semi-Supervised Learning
Our model family consists of a latent-variable generative model and a discriminative labeler.
Ranked #72 on
Named Entity Recognition (NER)
on CoNLL 2003 (English)
Chess as a Testbed for Language Model State Tracking
Motivated by this issue, we consider the task of language modeling for the game of chess.
Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks
Long document coreference resolution remains a challenging task due to the large memory and runtime requirements of current models.
Ranked #9 on
Coreference Resolution
on CoNLL 2012
SummScreen: A Dataset for Abstractive Screenplay Summarization
Since characters are fundamental to TV series, we also propose two entity-centric evaluation metrics.
On Generalization in Coreference Resolution
While coreference resolution is defined independently of dataset domain, most models for performing coreference resolution do not transfer well to unseen domains.
Ranked #1 on
Coreference Resolution
on WikiCoref
Smaller Text Classifiers with Discriminative Cluster Embeddings
Word embedding parameters often dominate overall model sizes in neural methods for natural language processing.
Learning Approximate Inference Networks for Structured Prediction
Prior work used gradient descent for inference, relaxing the structured output to a set of continuous variables and then optimizing the energy with respect to them.
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation
We propose to train a non-autoregressive machine translation model to minimize the energy defined by a pretrained autoregressive model.
EntEval: A Holistic Evaluation Benchmark for Entity Representations
Rich entity representations are useful for a wide class of problems involving entities.
How to Ask Better Questions? A Large-Scale Multi-Domain Dataset for Rewriting Ill-Formed Questions
We present a large-scale dataset for the task of rewriting an ill-formed natural language question to a well-formed one.
A Cross-Task Analysis of Text Span Representations
Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution.
TVStoryGen: A Dataset for Generating Stories with Character Descriptions
We introduce TVStoryGen, a story generation dataset that requires generating detailed TV show episode recaps from a brief summary and a set of documents describing the characters involved.
Ranked #1 on
Story Generation
on Fandom dev
WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
Datasets for data-to-text generation typically focus either on multi-domain, single-sentence generation or on single-domain, long-form generation.
Mining Knowledge for Natural Language Inference from Wikipedia Categories
Accurate lexical entailment (LE) and natural language inference (NLI) often require large quantities of costly annotations.
Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information
In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses.
Early Methods for Detecting Adversarial Images
Many machine learning classifiers are vulnerable to adversarial perturbations.
NatCat: Weakly Supervised Text Classification with Naturally Annotated Resources
We describe NatCat, a large-scale resource for text classification constructed from three data sources: Wikipedia, Stack Exchange, and Reddit.
Unsupervised Label Refinement Improves Dataless Text Classification
This reliance causes dataless classifiers to be highly sensitive to the choice of label descriptions and hinders the broader application of dataless classification in practice.
Ranked #3 on
Zero-Shot Text Classification
on AG News
Exemplar-Controllable Paraphrasing and Translation using Bitext
Our experimental results show that our models achieve competitive results on controlled paraphrase generation and strong performance on controlled machine translation.
Adjusting for Dropout Variance in Batch Normalization and Weight Initialization
We show how to adjust for the variance introduced by dropout with corrections to weight initialization and Batch Normalization, yielding higher accuracy.
PeTra: A Sparsely Supervised Memory Model for People Tracking
We propose PeTra, a memory-augmented neural network designed to track entities in its memory slots.
Ranked #1 on
Coreference Resolution
on GAP
(F1 metric)
Benchmarking Approximate Inference Methods for Neural Structured Prediction
One approach is to perform gradient descent with respect to the output structure directly (Belanger and McCallum, 2016).
The Benefits of Label-Description Training for Zero-Shot Text Classification
Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data in order to classify among specific label sets in downstream tasks.
An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks
Many tasks in natural language processing involve predicting structured outputs, e. g., sequence labeling, semantic role labeling, parsing, and machine translation.
Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing
We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers.
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations
We describe PARANMT-50M, a dataset of more than 50 million English-English sentential paraphrase pairs.
A Study of All-Convolutional Encoders for Connectionist Temporal Classification
We explore whether deep convolutional neural networks (CNNs) can be used effectively instead of RNNs as the "encoder" in CTC.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
End-to-End Neural Segmental Models for Speech Recognition
Segmental models are an alternative to frame-based models for sequence prediction, where hypothesized path weights are based on entire segment scores rather than a single frame at a time.
Learning to Embed Words in Context for Syntactic Tasks
We present models for embedding words in the context of surrounding words.
Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext
We consider the problem of learning general-purpose, paraphrastic sentence embeddings in the setting of Wieting et al. (2016b).
Emergent Predication Structure in Hidden State Vectors of Neural Readers
A significant number of neural architectures for reading comprehension have recently been developed and evaluated on large cloze-style datasets.
Revisiting Recurrent Networks for Paraphrastic Sentence Embeddings
We consider the problem of learning general-purpose, paraphrastic sentence embeddings, revisiting the setting of Wieting et al. (2016b).
Broad Context Language Modeling as Reading Comprehension
Progress in text understanding has been driven by large datasets that test particular capabilities, like recent datasets for reading comprehension (Hermann et al., 2015).
Ranked #32 on
Language Modelling
on LAMBADA
End-to-End Training Approaches for Discriminative Segmental Models
Similarly to hybrid HMM-neural network models, segmental models of this class can be trained in two stages (frame classifier training followed by linear segmental model weight training), end to end (joint training of both frame classifier and linear weights), or with end-to-end fine-tuning after two-stage training.
Who did What: A Large-Scale Person-Centered Cloze Dataset
We have constructed a new "Who-did-What" dataset of over 200, 000 fill-in-the-gap (cloze) multiple choice reading comprehension problems constructed from the LDC English Gigaword newswire corpus.
Discriminative Segmental Cascades for Feature-Rich Phone Recognition
A typical solution is to use approximate decoding, either by beam pruning in a single pass or by beam pruning to generate a lattice followed by a second pass.
Efficient Segmental Cascades for Speech Recognition
Discriminative segmental models offer a way to incorporate flexible feature functions into speech recognition.
Charagram: Embedding Words and Sentences via Character n-grams
We present Charagram embeddings, a simple approach for learning character-based compositional models to embed textual sequences.
Mapping Unseen Words to Task-Trained Embedding Spaces
We consider the supervised training setting in which we learn task-specific word embeddings.
Towards Universal Paraphrastic Sentence Embeddings
We again find that the word averaging models perform well for sentence similarity and entailment, outperforming LSTMs.
From Paraphrase Database to Compositional Paraphrase Model and Back
The Paraphrase Database (PPDB; Ganitkevitch et al., 2013) is an extensive semantic resource, consisting of a list of phrase pairs with (heuristic) confidence estimates.
Predicting the NFL using Twitter
We study the relationship between social media output and National Football League (NFL) games, using a dataset containing messages from Twitter and NFL game statistics.
Unsupervised Evaluation Metrics and Learning Criteria for Non-Parallel Textual Transfer
We show that the metric of post-transfer classification accuracy is insufficient on its own, and propose additional metrics based on semantic preservation and fluency as well as a way to combine them into a single overall score.
Pay Attention to the Ending:Strong Neural Baselines for the ROC Story Cloze Task
We consider the ROC story cloze task (Mostafazadeh et al., 2016) and present several findings.
Quality Signals in Generated Stories
We study the problem of measuring the quality of automatically-generated stories.
Constraints Based Convex Belief Propagation
Inference in Markov random fields subject to consistency structure is a fundamental problem that arises in many real-life applications.
A Sense-Topic Model for Word Sense Induction with Unsupervised Data Enrichment
Word sense induction (WSI) seeks to automatically discover the senses of a word in a corpus via unsupervised methods.
PoMo: Generating Entity-Specific Post-Modifiers in Context
Given a sentence about a target entity, the task is to automatically generate a post-modifier phrase that provides contextually relevant information about the entity.
Controllable Paraphrase Generation with a Syntactic Exemplar
Prior work on controllable text generation usually assumes that the controlled attribute can take on one of a small set of values known a priori.
Visually Grounded Neural Syntax Acquisition
We define concreteness of constituents by their matching scores with images, and use it to guide the parsing of text.
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity
While most neural machine translation (NMT)systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can significantly improve final translation accuracy.
Generating Diverse Story Continuations with Controllable Semantics
We propose a simple and effective modeling framework for controlled generation of multiple, diverse outputs.
Latent-Variable Generative Models for Data-Efficient Text Classification
This model consistently outperforms both the generative and discriminative classifiers in small-data settings.
Improving Joint Training of Inference Networks and Structured Prediction Energy Networks
Deep energy-based models are powerful, but pose challenges for learning and inference (Belanger and McCallum, 2016).
Learning Probabilistic Sentence Representations from Paraphrases
Probabilistic word embeddings have shown effectiveness in capturing notions of generality and entailment, but there is very little work on doing the analogous type of investigation for sentences.
Distractor Analysis and Selection for Multiple-Choice Cloze Questions for Second-Language Learners
Simple features of the distractor and correct answer correlate with the annotations, though we find substantial benefit to additionally using large-scale pretrained models to measure the fit of the distractor in the context.
Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
Learning Chess Blindfolded
Motivated by this issue, we consider the task of language modeling for the game of chess.
Adding Recurrence to Pretrained Transformers
Fine-tuning a pretrained transformer for a downstream task has become a standard method in NLP in the last few years.
On the Role of Supervision in Unsupervised Constituency Parsing
We analyze several recent unsupervised constituency parsing models, which are tuned with respect to the parsing $F_1$ score on the Wall Street Journal (WSJ) development set (1, 700 sentences).
Discriminatively-Tuned Generative Classifiers for Robust Natural Language Inference
We explore training objectives for discriminative fine-tuning of our generative classifiers, showing improvements over log loss fine-tuning from prior work .
Deep Clustering of Text Representations for Supervision-free Probing of Syntax
We explore deep clustering of text representations for unsupervised model interpretation and induction of syntax.
Substructure Substitution: Structured Data Augmentation for NLP
We study a family of data augmentation methods, substructure substitution (SUB2), for natural language processing (NLP) tasks.
FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models
Variational autoencoders (VAEs) are widely used for latent variable modeling of text.
Substructure Distribution Projection for Zero-Shot Cross-Lingual Dependency Parsing
We present substructure distribution projection (SubDP), a technique that projects a distribution over structures in one domain to another, by projecting substructure distributions separately.
Reconsidering the Past: Optimizing Hidden States in Language Models
We present Hidden-State Optimization (HSO), a gradient-based method for improving the performance of transformer language models at inference time.
"What makes a question inquisitive?" A Study on Type-Controlled Inquisitive Question Generation
We propose a type-controlled framework for inquisitive question generation.
“What makes a question inquisitive?” A Study on Type-Controlled Inquisitive Question Generation
We propose a type-controlled framework for inquisitive question generation.
Audio-Visual Neural Syntax Acquisition
We study phrase structure induction from visually-grounded speech.
MAP's not dead yet: Uncovering true language model modes by conditioning away degeneracy
It has been widely observed that exact or approximate MAP (mode-seeking) decoding from natural language generation (NLG) models consistently leads to degenerate outputs (Stahlberg and Byrne, 2019, Holtzman et al., 2019).
GEE! Grammar Error Explanation with Large Language Models
To address this gap, we propose the task of grammar error explanation, where a system needs to provide one-sentence explanations for each grammatical error in a pair of erroneous and corrected sentences.
