Search Results for author:
Found 47 papers, 27 papers with code
Evaluation Under Imperfect Benchmarks and Ratings: A Case Study in Text Simplification
Second, existing human ratings associated with the benchmarks often contain a high degree of disagreement, resulting in inconsistent ratings; nevertheless, existing metrics still have to show higher correlations with these imperfect ratings.
Robust Data Watermarking in Language Models by Injecting Fictitious Knowledge
Our watermarks are designed to be memorized by the LLM through seamlessly integrating in its training data, making them harder to detect lexically during preprocessing.
Political-LLM: Large Language Models in Political Science
In recent years, large language models (LLMs) have been widely adopted in political science tasks such as election prediction, sentiment analysis, policy impact assessment, and misinformation detection.
Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?
Subjective tasks in NLP have been mostly relegated to objective standards, where the gold label is decided by taking the majority vote.
Out-of-Distribution Detection through Soft Clustering with Non-Negative Kernel Regression
Our soft clustering approach for OOD detection highlights its potential for detecting tail-end phenomena in extreme-scale data settings.
Compare without Despair: Reliable Preference Evaluation with Generation Separability
We address these challenges by introducing a meta-evaluation measure, separability, which estimates how suitable a test instance is for pairwise preference evaluation.
OATH-Frames: Characterizing Online Attitudes Towards Homelessness with LLM Assistants
We release annotations with varying degrees of assistance from language models, with immense benefits in scaling: 6. 5x speedup in annotation time while only incurring a 3 point F1 reduction in performance with respect to the domain experts.
Annotating FrameNet via Structure-Conditioned Language Generation
Despite the remarkable generative capabilities of language models in producing naturalistic language, their effectiveness on explicit manipulation and generation of linguistic structures remain understudied.
Logits of API-Protected LLMs Leak Proprietary Information
Large language model (LLM) providers often hide the architectural details and parameters of their proprietary models by restricting public access to a limited API.
Closing the Curious Case of Neural Text Degeneration
We provide a theoretical explanation for the effectiveness of the truncation sampling by proving that truncation methods that discard tokens below some probability threshold (the most common type of truncation) can guarantee that all sampled tokens have nonzero true probability.
Does Video Summarization Require Videos? Quantifying the Effectiveness of Language in Video Summarization
Video summarization remains a huge challenge in computer vision due to the size of the input videos to be summarized.
COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements
To study the contextual dynamics of offensiveness, we train models to generate COBRA explanations, with and without access to the context.
NeuroComparatives: Neuro-Symbolic Distillation of Comparative Knowledge
We introduce NeuroComparatives, a novel framework for comparative knowledge distillation overgenerated from language models such as GPT-variants and LLaMA, followed by stringent filtering of the generated knowledge.
We're Afraid Language Models Aren't Modeling Ambiguity
We find that the task remains extremely challenging, including for GPT-4, whose generated disambiguations are considered correct only 32% of the time in human evaluation, compared to 90% for disambiguations in our dataset.
MAUVE Scores for Generative Models: Theory and Practice
We present MAUVE, a family of comparison measures between pairs of distributions such as those encountered in the generative modeling of text or images.
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation
Here, we investigate an alternative that a priori seems impossible: can smaller language models (e. g., GPT-2) win over models that are orders of magnitude larger and better (e. g., GPT-3), if powered with novel commonsense distillation algorithms?
NeuroCounterfactuals: Beyond Minimal-Edit Counterfactuals for Richer Data Augmentation
While counterfactual data augmentation offers a promising step towards robust generalization in natural language processing, producing a set of counterfactuals that offer valuable inductive bias for models remains a challenge.
REV: Information-Theoretic Evaluation of Free-Text Rationales
More concretely, we propose a metric called REV (Rationale Evaluation with conditional V-information), to quantify the amount of new, label-relevant information in a rationale beyond the information already available in the input or the label.
Investigating the Benefits of Free-Form Rationales
After controlling for instances where rationales leak the correct answer while not providing additional background knowledge, we find that incorporating only 5% of rationales during training can boost model performance by 47. 22% for CoS-E and 57. 14% for ECQA during inference.
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
Starting with an existing dataset, MultiNLI for natural language inference (NLI), our approach uses dataset cartography to automatically identify examples that demonstrate challenging reasoning patterns, and instructs GPT-3 to compose new examples with similar patterns.
Reframing Human-AI Collaboration for Generating Free-Text Explanations
We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop.
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases.
Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information
However, this comparison provides little understanding of how difficult each instance in a given distribution is, or what attributes make the dataset difficult for a given model.
Sister Help: Data Augmentation for Frame-Semantic Role Labeling
While FrameNet is widely regarded as a rich resource of semantics in natural language processing, a major criticism concerns its lack of coverage and the relative paucity of its labeled data compared to other commonly used lexical resources such as PropBank and VerbNet.
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text.
Contrastive Explanations for Model Interpretability
Our method is based on projecting model representation to a latent space that captures only the features that are useful (to the model) to differentiate two potential decisions.
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
As major progress is made in open-ended text generation, measuring how close machine-generated text is to human language remains a critical open problem.
Challenges in Automated Debiasing for Toxic Language Detection
Overall, our findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases.
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics
Experiments across four datasets show that these model-dependent measures reveal three distinct regions in the data map, each with pronounced characteristics.
Generative Data Augmentation for Commonsense Reasoning
Recent advances in commonsense reasoning depend on large-scale human-annotated training data to achieve peak performance.
Ranked #1 on
Question Answering
on CODAH
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
Language models pretrained on text from a wide variety of sources form the foundation of today's NLP.
The Right Tool for the Job: Matching Model and Instance Complexities
Our method presents a favorable speed/accuracy tradeoff in almost all cases, producing models which are up to five times faster than the state of the art, while preserving their accuracy.
Adversarial Filters of Dataset Biases
Large neural models have demonstrated human-level performance on language and vision benchmarks, while their performance degrades considerably on adversarial or out-of-distribution samples.
Shallow Syntax in Deep Water
Shallow syntax provides an approximation of phrase-syntactic structure of sentences; it can be produced with high accuracy, and is computationally cheap to obtain.
Transfer Learning in Natural Language Processing
The classic supervised machine learning paradigm is based on learning in isolation, a single predictive model for a task using a single dataset.
Syntactic Scaffolds for Semantic Structures
We introduce the syntactic scaffold, an approach to incorporating syntactic information into semantic tasks.
Polyglot Semantic Role Labeling
Previous approaches to multilingual semantic dependency parsing treat languages independently, without exploiting the similarities between semantic structures across languages.
Learning Joint Semantic Parsers from Disjoint Data
We present a new approach to learning semantic parsers from multiple datasets, even when the target semantic formalisms are drastically different, and the underlying corpora do not overlap.
Annotation Artifacts in Natural Language Inference Data
Large-scale datasets for natural language inference are created by presenting crowd workers with a sentence (premise), and asking them to generate three new sentences (hypotheses) that it entails, contradicts, or is logically neutral with respect to.
Multi-Mention Learning for Reading Comprehension with Neural Cascades
Reading comprehension is a challenging task, especially when executed across longer or across multiple evidence documents, where the answer is likely to reoccur.
Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold
We present a new, efficient frame-semantic parser that labels semantic arguments to FrameNet predicates.
DyNet: The Dynamic Neural Network Toolkit
In the static declaration strategy that is used in toolkits like Theano, CNTK, and TensorFlow, the user first defines a computation graph (a symbolic representation of the computation), and then examples are fed into an engine that executes this computation and computes its derivatives.
Greedy, Joint Syntactic-Semantic Parsing with Stack LSTMs
We present a transition-based parser that jointly produces syntactic and semantic dependencies.
