Search Results for author:
Found 256 papers, 121 papers with code
Shared Logistic Normal Distributions for Soft Parameter Tying in Unsupervised Grammar Induction
We present a family of priors over probabilistic grammar weights, called the shared logistic normal distribution.
Ranked #4 on
Unsupervised Dependency Parsing
on Penn Treebank
Dependency Grammar Induction
Unsupervised Dependency Parsing
Empirical Risk Minimization with Approximations of Probabilistic Grammars
Probabilistic grammars are generative statistical models that are useful for compositional and sequential structures.
Diffusion of Lexical Change in Social Media
Computer-mediated communication is driving fundamental changes in the nature of written language.
New Alignment Methods for Discriminative Book Summarization
We consider the unsupervised alignment of the full text of a book with a human-written summary.
A framework for (under)specifying dependency syntax without overloading annotators
We introduce a framework for lightweight dependency syntax annotation.
A Sparse and Adaptive Prior for Time-Dependent Model Parameters
We consider the scenario where the parameters of a probabilistic model are expected to vary over time.
Predicting the NFL using Twitter
We study the relationship between social media output and National Football League (NFL) games, using a dataset containing messages from Twitter and NFL game statistics.
Dynamic Language Models for Streaming Text
We present a probabilistic language model that captures temporal dynamics and conditions on arbitrary non-linguistic context features.
Unsupervised Discovery of Biographical Structure from Text
We present a method for discovering abstract event classes in biographies, based on a probabilistic latent-variable model.
Discriminative Lexical Semantic Segmentation with Gaps: Running the MWE Gamut
We present a novel representation, evaluation measure, and supervised models for the task of identifying the multiword expressions (MWEs) in a sentence, resulting in a lexical semantic segmentation.
An Empirical Comparison of Parsing Methods for Stanford Dependencies
Stanford typed dependencies are a widely desired representation of natural language sentences, but parsing is one of the major computational bottlenecks in text analysis systems.
Comprehensive Annotation of Multiword Expressions in a Social Web Corpus
Multiword expressions (MWEs) are quite frequent in languages such as English, but their diversity, the scarcity of individual MWE types, and contextual ambiguity have presented obstacles to corpus-based studies and NLP systems addressing them as a class.
Learning Word Representations with Hierarchical Sparse Coding
We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings.
The Utility of Text: The Case of Amicus Briefs and the Supreme Court
We explore the idea that authoring a piece of text is an act of maximizing one's expected utility.
Conditional Random Field Autoencoders for Unsupervised Structured Prediction
We introduce a framework for unsupervised learning of structured predictors with overlapping, global features.
Retrofitting Word Vectors to Semantic Lexicons
Vector space word representations are learned from distributional information of words in large corpora.
Bayesian Optimization of Text Representations
When applying machine learning to problems in NLP, there are many choices to make about how to represent input texts.
Transition-Based Dependency Parsing with Stack Long Short-Term Memory
We propose a technique for learning representations of parser states in transition-based dependency parsers.
Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs
We present extensions to a continuous-state dependency parsing method that makes it applicable to morphologically rich languages.
Segmental Recurrent Neural Networks
Representations of the input segments (i. e., contiguous subsequences of the input) are computed by encoding their constituent tokens using bidirectional recurrent neural nets, and these "segment embeddings" are used to define compatibility scores with output labels.
Annotating Character Relationships in Literary Texts
We present a dataset of manually annotated relationships between characters in literary texts, in order to support the training and evaluation of automatic methods for relation type prediction in this domain (Makazhanov et al., 2014; Kokkinakis, 2013) and the broader computational analysis of literary character (Elson et al., 2010; Bamman et al., 2014; Vala et al., 2015; Flekova and Gurevych, 2015).
Many Languages, One Parser
We train one multilingual model for dependency parsing and use it to parse sentences in several languages.
Massively Multilingual Word Embeddings
We introduce new methods for estimating and evaluating embeddings of words in more than fifty languages in a single shared embedding space.
Recurrent Neural Network Grammars
We introduce recurrent neural network grammars, probabilistic models of sentences with explicit phrase structure.
Ranked #25 on
Constituency Parsing
on Penn Treebank
Segmental Recurrent Neural Networks for End-to-end Speech Recognition
This model connects the segmental conditional random field (CRF) with a recurrent neural network (RNN) used for feature extraction.
Ranked #16 on
Speech Recognition
on TIMIT
Training with Exploration Improves a Greedy Stack-LSTM Parser
We adapt the greedy Stack-LSTM dependency parser of Dyer et al. (2015) to support a training-with-exploration procedure using dynamic oracles(Goldberg and Nivre, 2013) instead of cross-entropy minimization.
Ranked #2 on
Chinese Dependency Parsing
on Chinese Pennbank
Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs
We present a transition-based parser that jointly produces syntactic and semantic dependencies.
Hierarchical Character-Word Models for Language Identification
Social media messages' brevity and unconventional spelling pose a challenge to language identification.
Distilling an Ensemble of Greedy Dependency Parsers into One MST Parser
We introduce two first-order graph-based dependency parsers achieving a new state of the art.
Ranked #17 on
Dependency Parsing
on Penn Treebank
Character Sequence Models for ColorfulWords
We present a neural network architecture to predict a point in color space from the sequence of characters in the color's name.
What Do Recurrent Neural Network Grammars Learn About Syntax?
We investigate what information they learn, from a linguistic perspective, through various ablations to the model and the data, and by augmenting the model with an attention mechanism (GA-RNNG) to enable closer inspection.
Ranked #20 on
Constituency Parsing
on Penn Treebank
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
A writer's style depends not just on personal traits but also on her intent and mental state.
Multitask Learning with CTC and Segmental CRF for Speech Recognition
Segmental conditional random fields (SCRFs) and connectionist temporal classification (CTC) are two sequence labeling methods used for end-to-end training of speech recognition models.
Story Cloze Task: UW NLP System
This paper describes University of Washington NLP{'}s submission for the Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem 2017) shared task{---}the Story Cloze Task.
Deep Multitask Learning for Semantic Dependency Parsing
We present a deep neural architecture that parses sentences into three semantic dependency graph formalisms.
Friendships, Rivalries, and Trysts: Characterizing Relations between Ideas in Texts
Combining two statistics --- cooccurrence within documents and prevalence correlation over time --- our approach reveals a number of different ways in which ideas can cooperate and compete.
Neural Models for Documents with Metadata
Most real-world document collections involve various types of metadata, such as author, source, and date, and yet the most commonly-used approaches to modeling text corpora ignore this information.
Greedy Transition-Based Dependency Parsing with Stack LSTMs
During training, dynamic oracles alternate between sampling parser states from the training data and from the model as it is being learned, making the model more robust to the kinds of errors that will be made at test time.
Open Loop Hyperparameter Optimization and Determinantal Point Processes
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated.
Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold
We present a new, efficient frame-semantic parser that labels semantic arguments to FrameNet predicates.
End-to-End Neural Segmental Models for Speech Recognition
Segmental models are an alternative to frame-based models for sequence prediction, where hypothesized path weights are based on entire segment scores rather than a single frame at a time.
Dynamic Entity Representations in Neural Language Models
Understanding a long document requires tracking how entities are introduced and evolve over time.
"You are no Jack Kennedy": On Media Selection of Highlights from Presidential Debates
We first examine the effect of wording and propose a binary classification framework that controls for both the speaker and the debate situation.
Annotation Artifacts in Natural Language Inference Data
Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to.
Learning Joint Semantic Parsers from Disjoint Data
We present a new approach to learning semantic parsers from multiple datasets, even when the target semantic formalisms are drastically different, and the underlying corpora do not overlap.
Parsing Tweets into Universal Dependencies
Nonetheless, using the new treebank, we build a pipeline system to parse raw tweets into UD.
Ranked #2 on
Dependency Parsing
on Tweebank
Sounding Board: A User-Centric and Content-Driven Social Chatbot
We present Sounding Board, a social chatbot that won the 2017 Amazon Alexa Prize.
Backpropagating through Structured Argmax using a SPIGOT
We introduce the structured projection of intermediate gradients optimization technique (SPIGOT), a new method for backpropagating through neural networks that include hard-decision structured predictions (e. g., parsing) in intermediate layers.
SoPa: Bridging CNNs, RNNs, and Weighted Finite-State Machines
Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances.
Explainable artificial intelligence
General Classification
+3
Event2Mind: Commonsense Inference on Events, Intents, and Reactions
We investigate a new commonsense inference task: given an event described in a short free-form text ("X drinks coffee in the morning"), a system reasons about the likely intents ("X wants to stay awake") and reactions ("X feels alert") of the event's participants.
Ranked #1 on
Common Sense Reasoning
on Event2Mind test
Toward Abstractive Summarization Using Semantic Representations
We present a novel abstractive summarization framework that draws on the recent development of a treebank for the Abstract Meaning Representation (AMR).
LSTMs Exploit Linguistic Attributes of Data
While recurrent neural networks have found success in a variety of natural language processing applications, they are general models of sequential data.
Discovering Phonesthemes with Sparse Regularization
We introduce a simple method for extracting non-arbitrary form-meaning representations from a collection of semantic vectors.
The Importance of Calibration for Estimating Proportions from Annotations
Estimating label proportions in a target corpus is a type of measurement that is useful for answering certain types of social-scientific questions.
Neural Text Generation in Stories Using Entity Representations as Context
We introduce an approach to neural text generation that explicitly represents entities mentioned in the text.
Bridging CNNs, RNNs, and Weighted Finite-State Machines
Recurrent and convolutional neural networks comprise two distinct families of models that have proven to be useful for encoding natural language utterances.
Semantic Matching Against a Corpus: New Applications and Methods
We consider the case of a domain expert who wishes to explore the extent to which a particular idea is expressed in a text collection.
Rational Recurrences
We characterize this connection formally, defining rational recurrences to be recurrent hidden state update functions that can be written as the Forward calculation of a finite set of WFSAs.
Neural Cross-Lingual Named Entity Recognition with Minimal Resources
To improve robustness to word order differences, we propose to use self-attention, which allows for a degree of flexibility with respect to word order.
Syntactic Scaffolds for Semantic Structures
We introduce the syntactic scaffold, an approach to incorporating syntactic information into semantic tasks.
ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
We present ATOMIC, an atlas of everyday commonsense reasoning, organized through 877k textual descriptions of inferential knowledge.
You May Not Need Attention
In NMT, how far can we get without attention and without separate encoding and decoding?
Deep Weighted Averaging Classifiers
Recent advances in deep learning have achieved impressive gains in classification accuracy on a variety of types of data, including images and text.
Contextual Word Representations: A Contextual Introduction
This introduction aims to tell the story of how we put words into computers.
Polyglot Contextual Representations Improve Crosslingual Transfer
We introduce Rosita, a method to produce multilingual contextual word representations by training a single language model on text from multiple languages.
Measuring Online Debaters' Persuasive Skill from Text over Time
We build on a widely used model of skill in two-player games and augment it with linguistic features of a debater{'}s content.
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
While most previous work has focused on different pretraining objectives and architectures for transfer learning, we ask how to best adapt the pretrained model to a given target task.
Linguistic Knowledge and Transferability of Contextual Representations
Contextual word representations derived from large-scale neural language models are successful across a diverse set of NLP tasks, suggesting that they encode useful and transferable features of language.
Inoculation by Fine-Tuning: A Method for Analyzing Challenge Datasets
Several datasets have recently been constructed to expose brittleness in models trained on existing benchmarks.
Evaluating Gender Bias in Machine Translation
We present the first challenge set and evaluation protocol for the analysis of gender bias in machine translation (MT).
Variational Pretraining for Semi-supervised Text Classification
We accompany this paper with code to pretrain and use VAMPIRE embeddings in downstream tasks.
Is Attention Interpretable?
Attention mechanisms have recently boosted performance on a range of NLP tasks.
The Risk of Racial Bias in Hate Speech Detection
We investigate how annotators{'} insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations.
Sentence Mover's Similarity: Automatic Evaluation for Multi-Sentence Texts
For evaluating machine-generated texts, automatic methods hold the promise of avoiding collection of human judgments, which can be expensive and time-consuming.
Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
Machine comprehension of texts longer than a single sentence often requires coreference resolution.
Shallow Syntax in Deep Water
Shallow syntax provides an approximation of phrase-syntactic structure of sentences; it can be produced with high accuracy, and is computationally cheap to obtain.
Topics to Avoid: Demoting Latent Confounds in Text Classification
Despite impressive performance on many text classification tasks, deep neural networks tend to learn frequent superficial patterns that are specific to the training data and do not always generalize well.
PaLM: A Hybrid Parser and Language Model
We present PaLM, a hybrid parser and neural language model.
RNN Architecture Learning with Sparse Regularization
Our method also highlights the interpretable properties of rational RNNs.
Show Your Work: Improved Reporting of Experimental Results
Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e. g., accuracy) on held-out test data, compared to previous results.
Knowledge Enhanced Contextual Word Representations
Contextual word representations, typically trained on unstructured, unlabeled text, do not contain any explicit grounding to real world entities and are often unable to remember facts about those entities.
Ranked #9 on
Relation Classification
on TACRED
Improving Natural Language Inference with a Pretrained Parser
We introduce a novel approach to incorporate syntax into natural language inference (NLI) models.
Low-Resource Parsing with Crosslingual Contextualized Representations
Despite advances in dependency parsing, languages with small treebanks still present challenges.
Situating Sentence Embedders with Nearest Neighbor Overlap
As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words have come to play an increasingly important role.
Social Bias Frames: Reasoning about Social and Power Implications of Language
We introduce Social Bias Frames, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others.
Improving Transformer Models by Reordering their Sublayers
Multilayer transformer networks consist of interleaved self-attention and feedforward sublayers.
Ranked #7 on
Language Modelling
on enwik8
On Consequentialism and Fairness
In this paper we provide a consequentialist critique of common definitions of fairness within machine learning, as well as a machine learning perspective on consequentialism.
Explaining Relationships Between Scientific Documents
We address the task of explaining relationships between two scientific documents using natural language text.
Multi-View Learning for Vision-and-Language Navigation
Learning to navigate in a visual environment following natural language instructions is a challenging task because natural language instructions are highly variable, ambiguous, and under-specified.
Evaluating Models' Local Decision Boundaries via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
The Right Tool for the Job: Matching Model and Instance Complexities
Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy.
A Formal Hierarchy of RNN Architectures
While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy.
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP.
A Mixture of $h-1$ Heads is Better than $h$ Heads
Multi-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks.
Multilingual and Interlingual Semantic Representations for Natural Language Processing: A Brief Introduction
We introduce the Computational Linguistics special issue on Multilingual and Interlingual Semantic Representations for Natural Language Processing.
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
We show that the speed disadvantage for autoregressive baselines compared to non-autoregressive methods has been overestimated in three aspects: suboptimal layer allocation, insufficient speed measurement, and lack of knowledge distillation.
Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus.
We analyze the effect of author and reader characteristics and story writing setup on the quality of stories in a short storytelling task.
A Mixture of h - 1 Heads is Better than h Heads
Multi-head attentive neural architectures have achieved state-of-the-art results on a variety of natural language processing tasks.
Recollection versus Imagination: Exploring Human Memory and Cognition via Neural Language Models
We introduce a measure of narrative flow and use this to examine the narratives for imagined and recalled events.
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics.
Grounded Compositional Outputs for Adaptive Language Modeling
To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank
Pretrained multilingual contextual representations have shown great success, but due to the limits of their pretraining data, their benefits do not apply equally to all language varieties.
Evaluating NLP Models via Contrast Sets
Unfortunately, when a dataset has systematic gaps (e. g., annotation artifacts), these evaluations are misleading: a model can learn simple decision rules that perform well on the test set but do not capture a dataset's intended capabilities.
Multilevel Text Alignment with Cross-Document Attention
Text alignment finds application in tasks such as citation recommendation and plagiarism detection.
Plug and Play Autoencoders for Conditional Text Generation
Text autoencoders are commonly used for conditional generation tasks such as style transfer.
The Multilingual Amazon Reviews Corpus
We present the Multilingual Amazon Reviews Corpus (MARC), a large-scale collection of Amazon reviews for multilingual text classification.
Unsupervised Bitext Mining and Translation via Self-trained Contextual Embeddings
We then improve an XLM-based unsupervised neural MT system pre-trained on Wikipedia by supplementing it with pseudo-parallel text mined from the same corpus, boosting unsupervised translation performance by up to 3. 5 BLEU on the WMT'14 French-English and WMT'16 German-English tasks and outperforming the previous state-of-the-art.
Natural Language Rationales with Full-Stack Visual Reasoning: From Pixels to Semantic Frames to Commonsense Graphs
Natural language rationales could provide intuitive, higher-level explanations that are easily understandable by humans, complementing the more broadly studied lower-level explanations based on gradients or attention weights.
Measuring Association Between Labels and Free-Text Rationales
In interpretable NLP, we require faithful rationales that reflect the model's decision-making process for an explained instance.
Thinking Like a Skeptic: Defeasible Inference in Natural Language
Defeasible inference is a mode of reasoning in which an inference (X is a bird, therefore X flies) may be weakened or overturned in light of new evidence (X is a penguin).
Infusing Finetuning with Semantic Dependencies
For natural language processing systems, two kinds of evidence support the use of text representations from neural language models "pretrained" on large unannotated corpora: performance on application-inspired benchmarks (Peters et al., 2018, inter alia), and the emergence of syntactic abstractions in those representations (Tenney et al., 2019, inter alia).
Shortformer: Better Language Modeling using Shorter Inputs
Increasing the input length has been a driver of progress in language modeling with transformers.
Ranked #26 on
Language Modelling
on WikiText-103
GENIE: Toward Reproducible and Standardized Human Evaluation for Text Generation
While often assumed a gold standard, effective human evaluation of text generation remains an important, open area for research.
Challenges in Automated Debiasing for Toxic Language Detection
Overall, our findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases.
Random Feature Attention
RFA can be used as a drop-in replacement for conventional softmax attention and offers a straightforward way of learning with recency bias through an optional gating mechanism.
Ranked #27 on
Machine Translation
on IWSLT2014 German-English
Finetuning Pretrained Transformers into RNNs
Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune.
Ranked #2 on
Machine Translation
on WMT2017 Chinese-English
Probing Across Time: What Does RoBERTa Know and When?
Models of language trained on very large corpora have been demonstrated useful for NLP.
Competency Problems: On Finding and Removing Artifacts in Language Data
In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems.
Go Forth and Prosper: Language Modeling with Ancient Textual History
This method can improve perplexity of pretrained LMs with no updates to the LM's own parameters.
Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?
We study whether assertions enable a system to emulate representations preserving semantic relations like equivalence.
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text.
A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
Readers of academic research papers often read with the goal of answering specific questions.
Ranked #1 on
Evidence Selection
on QASPER
Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models
Story generation is an open-ended and subjective task, which poses a challenge for evaluating story generation models.
Specializing Multilingual Language Models: An Empirical Study
Pretrained multilingual language models have become a common tool in transferring NLP capabilities to low-resource languages, often with adaptations.
Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study
Biomedical knowledge graphs (KGs) hold rich information on entities such as diseases, drugs, and genes.
Saturated Transformers are Constant-Depth Threshold Circuits
Transformers have become a standard neural network architecture for many NLP problems, motivating theoretical analysis of their power in terms of formal languages.
All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Human evaluations are typically considered the gold standard in natural language generation, but as models' fluency improves, how well can evaluators detect and judge machine-generated text?
Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text
To support the broad range of real machine errors that can be identified by laypeople, the ten error categories of Scarecrow -- such as redundancy, commonsense errors, and incoherence -- are identified through several rounds of crowd annotation experiments without a predefined ontology.
All That's `Human' Is Not Gold: Evaluating Human Evaluation of Generated Text
Human evaluations are typically considered the gold standard in natural language generation, but as models{'} fluency improves, how well can evaluators detect and judge machine-generated text?
DEMix Layers: Disentangling Domains for Modular Language Modeling
We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text.
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
Since the introduction of the transformer model by Vaswani et al. (2017), a fundamental question has yet to be answered: how does a model achieve extrapolation at inference time for sequences that are longer than it saw during training?
Sentence Bottleneck Autoencoders from Transformer Language Models
Representation learning for text via pretraining a language model on a large corpus has become a standard starting point for building NLP systems.
Expected Validation Performance and Estimation of a Random Variable's Maximum
We find that the two biased estimators lead to the fewest incorrect conclusions, which hints at the importance of minimizing variance and MSE.
ABC: Attention with Bounded-memory Control
One way to improve the efficiency is to bound the memory size.
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance.
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases.
Transparent Human Evaluation for Image Captioning
We establish THumB, a rubric-based human evaluation protocol for image captioning models.
Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand
We therefore propose a generalization of leaderboards, bidimensional leaderboards (Billboards), that simultaneously tracks progress in language generation models and metrics for their evaluation.
NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics
To enable constrained generation, we build on NeuroLogic decoding (Lu et al., 2021), combining its flexibility in incorporating logical constraints with A*esque estimates of future constraint satisfaction.
Ranked #1 on
Text Generation
on ROCStories
Imagined versus Remembered Stories: Quantifying Differences in Narrative Flow
We quantify the differences between autobiographical and imagined stories by introducing sequentiality, a measure of narrative flow of events, drawing probabilistic inferences from a cutting-edge large language model (GPT-3).
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns.
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models
Structured knowledge grounding (SKG) leverages structured knowledge to complete user requests, such as semantic parsing over databases and question answering over knowledge bases.
Ranked #1 on
Task-Oriented Dialogue Systems
on KVRET
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection
Language models increasingly rely on massive web dumps for diverse text data.
In-Context Learning for Few-Shot Dialogue State Tracking
In this work, we propose an in-context learning (ICL) framework for zero-shot and few-shot learning DST, where a large pre-trained language model (LM) takes a test instance and a few exemplars as input, and directly decodes the dialogue state without any parameter updates.
A Call for Clarity in Beam Search: How It Works and When It Stops
Based on this finding, we introduce a patience factor, a simple modification to this beam decoding implementation, that generalizes the stopping criterion and provides flexibility to the depth of search.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
