Search Results for author:
Found 132 papers, 65 papers with code
Belief in the Machine: Investigating Epistemological Blind Spots of Language Models
These findings highlight significant concerns about current LMs' ability to reason about truth, belief, and knowledge while emphasizing the need for advancements in these areas before broad deployment in critical sectors.
Bayesian scaling laws for in-context learning
In all cases, Bayesian scaling laws accurately predict the conditions under which ICL will cause the suppressed behavior to reemerge, which sheds light on the ineffectiveness of post-training at increasing LLM safety.
Can Unconfident LLM Annotations Be Used for Confident Conclusions?
We introduce Confidence-Driven Inference: a method that combines LLM annotations and LLM confidence indicators to strategically select which human annotations should be collected, with the goal of producing accurate statistical estimates and provably valid confidence intervals while reducing the number of human annotations needed.
A layer-wise analysis of Mandarin and English suprasegmentals in SSL speech models
This study asks how self-supervised speech models represent suprasegmental categories like Mandarin lexical tone, English lexical stress, and English phrasal accents.
h4rm3l: A Dynamic Benchmark of Composable Jailbreak Attacks for LLM Safety Assessment
The safety of Large Language Models (LLMs) remains a critical concern due to a lack of adequate benchmarks for systematically evaluating their ability to resist generating harmful content.
Data Checklist: On Unit-Testing Datasets with Usable Information
Model checklists (Ribeiro et al., 2020) have emerged as a useful tool for understanding the behavior of LLMs, analogous to unit-testing in software engineering.
Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance
The ability to communicate uncertainty, risk, and limitation is crucial for the safety of large language models.
ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets
This paper presents ML-SUPERB~2. 0, which is a new benchmark for evaluating pre-trained SSL and supervised speech models across downstream models, fine-tuning setups, and efficient model adaptation approaches.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+4
ReFT: Representation Finetuning for Language Models
We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency.
NLP Systems That Can't Tell Use from Mention Censor Counterspeech, but Teaching the Distinction Helps
The use of words to convey speaker's intent is traditionally distinguished from the `mention' of words for quoting what someone said, or pointing out properties of a word.
Dialect prejudice predicts AI decisions about people's character, employability, and criminality
Here, we demonstrate that language models embody covert racism in the form of dialect prejudice: we extend research showing that Americans hold raciolinguistic stereotypes about speakers of African American English and find that language models have the same prejudice, exhibiting covert stereotypes that are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to the ones from before the civil rights movement.
CausalGym: Benchmarking causal interpretability methods on linguistic tasks
Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e. g., surprisal comparisons).
How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis
We develop NegotiationArena: a flexible framework for evaluating and probing the negotiation abilities of LLM agents.
AnthroScore: A Computational Linguistic Measure of Anthropomorphism
Anthropomorphism, or the attribution of human-like characteristics to non-human entities, has shaped conversations about the impacts and possibilities of technology.
Predicting positive transfer for improved low-resource speech recognition using acoustic pseudo-tokens
Continued pre-training on 70-200 hours of untranscribed speech in these languages can help -- but what about languages without that much recorded data?
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
KTO: Model Alignment as Prospect Theoretic Optimization
Kahneman & Tversky's $\textit{prospect theory}$ tells us that humans perceive random variables in a biased but well-defined manner (1992); for example, humans are famously loss-averse.
Multilingual self-supervised speech representations improve the speech recognition of low-resource African languages with codeswitching
While many speakers of low-resource languages regularly code-switch between their languages and other regional languages or English, datasets of codeswitched speech are too small to train bespoke acoustic models from scratch or do language model rescoring.
Grounding Gaps in Language Model Generations
To understand the roots of the identified grounding gap, we examine the role of instruction tuning and preference optimization, finding that training on contemporary preference data leads to a reduction in generated grounding acts.
A Benchmark for Learning to Translate a New Language from One Grammar Book
In this paper, we introduce MTOB (Machine Translation from One Book), a benchmark for learning to translate between English and Kalamang -- a language with less than 200 speakers and therefore virtually no presence on the web -- using several hundred pages of field linguistics reference materials.
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful.
Learning the meanings of function words from grounded language using a visual question answering model
Furthermore, we find that these models can learn the meanings of logical connectives and and or without any prior knowledge of logical reasoning, as well as early evidence that they are sensitive to alternative expressions when interpreting language.
Othering and low status framing of immigrant cuisines in US restaurant reviews and large language models
Through careful linguistic analyses, we evaluate social theories about attitudes toward immigrant cuisine in a large-scale study of framing differences in 2. 1M English language Yelp reviews.
Ecosystem-level Analysis of Deployed Machine Learning Reveals Homogeneous Outcomes
In practice, the societal impact of machine learning is determined by the surrounding context of machine learning deployments.
Developing Speech Processing Pipelines for Police Accountability
We investigate the potential of large pre-trained speech models for facilitating reviews, focusing on ASR and officer speech detection in footage from traffic stops.
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models
To recognize and mitigate harms from large language models (LLMs), we need to understand the prevalence and nuances of stereotypes in LLM outputs.
Making More of Little Data: Improving Low-Resource Automatic Speech Recognition Using Data Augmentation
For Gronings, for which there was a pre-existing text-to-speech (TTS) system available, we also examined the use of TTS to generate ASR training data from text-only sources.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+3
string2string: A Modern Python Library for String-to-String Algorithms
It includes traditional algorithmic solutions as well as recent advanced neural approaches to tackle various problems in string alignment, distance measurement, lexical and semantic search, and similarity analysis -- along with several helpful visualization tools and metrics to facilitate the interpretation and analysis of these methods.
Injecting structural hints: Using language models to study inductive biases in language learning
Our study leverages the capabilities of transformer models to run controlled language learning experiments that are not possible to run on humans, and surfaces hypotheses about the structures that facilitate language learning in both humans and machines.
Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models
The increased deployment of LMs for real-world tasks involving knowledge and facts makes it important to understand model epistemology: what LMs think they know, and how their attitudes toward that knowledge are affected by language use in their inputs.
Leveraging supplementary text data to kick-start automatic speech recognition system development with limited transcriptions
We perform experiments using 10 minutes of transcribed speech from English (for replicating prior work) and two additional pairs of languages differing in the availability of supplemental text data: Gronings and Frisian (~7. 5M token corpora available), and Besemah and Nasal (only small lexica available).
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models
A growing ecosystem of large, open-source foundation models has reduced the labeled data and technical expertise necessary to apply machine learning to many new problems.
Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome Homogenization?
As the scope of machine learning broadens, we observe a recurring theme of algorithmic monoculture: the same systems, or systems that share components (e. g. training data), are deployed by multiple decision-makers.
Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decoding
In open-ended natural-language generation, existing text decoding methods typically struggle to produce text which is both diverse and high-quality.
Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scale
For example, we find cases of prompting for basic traits or social roles resulting in images reinforcing whiteness as ideal, prompting for occupations resulting in amplification of racial and gender disparities, and prompting for objects resulting in reification of American norms.
Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models
Here we show another problem with multilingual models: grammatical structures in higher-resource languages bleed into lower-resource languages, a phenomenon we call grammatical structure bias.
When and why vision-language models behave like bags-of-words, and what to do about it?
ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO & Flickr30k-Order, to test for order sensitivity.
Computationally Identifying Funneling and Focusing Questions in Classroom Discourse
Responsive teaching is a highly effective strategy that promotes student learning.
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
One concern with the rise of large language models lies with their potential for significant harm, particularly from pretraining on biased, obscene, copyrighted, and private information.
Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models
We propose a method for arbitrary textual style transfer (TST)--the task of transforming a text into any given style--utilizing general-purpose pre-trained language models.
Richer Countries and Richer Representations
We examine whether some countries are more richly represented in embedding space than others.
Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words
Cosine similarity of contextual embeddings is used in many NLP tasks (e. g., QA, IR, MT) and metrics (e. g., BERTScore).
Modular Domain Adaptation
Off-the-shelf models are widely used by computational social science researchers to measure properties of text, such as sentiment.
Automated speech tools for helping communities process restricted-access corpora for language revival efforts
An even narrower bottleneck occurs for recordings with access constraints, such as language that must be vetted or filtered by authorised community members before annotation can begin.
Focus on what matters: Applying Discourse Coherence Theory to Cross Document Coreference
We model the entities/events in a reader's focus as a neighborhood within a learned latent embedding space which minimizes the distance between mentions and the centroids of their gold coreference clusters.
Ranked #1 on
Event Coreference Resolution
on Gun Violence Corpus
coreference-resolution
Entity Cross-Document Coreference Resolution
+2
The Emergence of the Shape Bias Results from Communicative Efficiency
By the age of two, children tend to assume that new word categories are based on objects' shape, rather than their color or texture; this assumption is called the shape bias.
On the Opportunities and Risks of Foundation Models
AI is undergoing a paradigm shift with the rise of models (e. g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks.
Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions
In conversation, uptake happens when a speaker builds on the contribution of their interlocutor by, for example, acknowledging, repeating or reformulating what they have said.
Attention Flows are Shapley Value Explanations
Shapley Values, a solution to the credit assignment problem in cooperative game theory, are a popular type of explanation in machine learning, having been used to explain the importance of features, embeddings, and even neurons.
Sensitivity as a Complexity Measure for Sequence Classification Tasks
We introduce a theoretical framework for understanding and predicting the complexity of sequence classification tasks, using a novel extension of the theory of Boolean function sensitivity.
Frequency-based Distortions in Contextualized Word Embeddings
How does word frequency in pre-training data affect the behavior of similarity metrics in contextualized BERT embeddings?
Leveraging pre-trained representations to improve access to untranscribed speech from endangered languages
Surprisingly, the English model outperformed the multilingual model on 4 Australian language datasets, raising questions around how to optimally leverage self-supervised speech representations for QbE-STD.
Automatic Speech Recognition
Automatic Speech Recognition (ASR)
+1
Language Through a Prism: A Spectral Approach for Multiscale Language Representations
Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents.
Detecting Stance in Media on Global Warming
We release our stance dataset, model, and lexicons of framing devices for future work on opinion-framing and the automatic detection of GW stance.
Causal Effects of Linguistic Properties
Second, in practice, we only have access to noisy proxies for the linguistic properties of interest -- e. g., predictions from classifiers and lexicons.
Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
We further show via a human evaluation and a qualitative analysis that our system leads to generations that are more factually complete and consistent compared to the baselines.
With Little Power Comes Great Responsibility
Despite its importance to experimental design, statistical power (the probability that, given a real effect, an experiment will reject the null hypothesis) has largely been ignored by the NLP community.
Nearest Neighbor Machine Translation
We introduce $k$-nearest-neighbor machine translation ($k$NN-MT), which predicts tokens with a nearest neighbor classifier over a large datastore of cached examples, using representations from a neural translation model for similarity search.
Utility is in the Eye of the User: A Critique of NLP Leaderboards
Benchmarks such as GLUE have helped drive advances in NLP by incentivizing the creation of more accurate models.
The Role of Verb Semantics in Hungarian Verb-Object Order
We test the effect of lexical semantics on the ordering of verbs and their objects by grouping verbs into 11 semantic classes.
Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models
Recent models for unsupervised representation learning of text have employed a number of techniques to improve contextual word representations but have put little focus on discourse-level representations.
Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models
We propose transfer learning as a method for analyzing the encoding of grammatical structure in neural language models.
A Framework for the Computational Linguistic Analysis of Dehumanization
Dehumanization is a pernicious psychological process that often leads to extreme intergroup bias, hate speech, and violence aimed at targeted social groups.
Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learning
Accurate reporting of energy and carbon usage is essential for understanding the potential climate impacts of machine learning research.
Social Bias Frames: Reasoning about Social and Power Implications of Language
We introduce Social Bias Frames, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others.
Generalization through Memorization: Nearest Neighbor Language Models
Applying this augmentation to a strong Wikitext-103 LM, with neighbors drawn from the original training set, our $k$NN-LM achieves a new state-of-the-art perplexity of 15. 79 - a 2. 9 point improvement with no additional training.
Ranked #10 on
Language Modelling
on WikiText-103
The Diversity-Innovation Paradox in Science
Are underrepresented groups more likely to generate scientific innovations?
From Insanely Jealous to Insanely Delicious: Computational Models for the Semantic Bleaching of English Intensifiers
We introduce novel computational models for modeling semantic bleaching, a widespread category of change in which words become more abstract or lose elements of meaning, like the development of {``}arrive{''} from its earlier meaning {`}become at shore.
Neural Text Style Transfer via Denoising and Reranking
We introduce a simple method for text style transfer that frames style transfer as denoising: we synthesize a noisy corpus and treat the source style as a noisy version of the target style.
Recursive Routing Networks: Learning to Compose Modules for Language Understanding
The model jointly optimizes the parameters of the functions and the meta-learner{'}s policy for routing inputs through those functions.
Let's Make Your Request More Persuasive: Modeling Persuasive Strategies via Semi-Supervised Neural Nets on Crowdfunding Platforms
Modeling what makes a request persuasive - eliciting the desired response from a reader - is critical to the study of propaganda, behavioral economics, and advertising.
Sample Efficient Text Summarization Using a Single Pre-Trained Transformer
Language model (LM) pre-training has resulted in impressive performance and sample efficiency on a variety of language understanding tasks.
Ranked #1 on
Text Summarization
on DUC 2004 Task 1
(ROUGE-2 metric)
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image.
Analyzing Polarization in Social Media: Method and Application to Tweets on 21 Mass Shootings
We provide an NLP framework to uncover four linguistic dimensions of political polarization in social media: topic choice, framing, affect and illocutionary force.
Textual Analogy Parsing: What's Shared and What's Compared among Analogous Facts
To understand a sentence like "whereas only 10% of White Americans live at or below the poverty line, 28% of African Americans do" it is important not only to identify individual facts, e. g., poverty rates of distinct demographic groups, but also the higher-order relations between them, e. g., the disparity between them.
Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategies
Amidst growing concern over media manipulation, NLP attention has focused on overt strategies like censorship and "fake news'".
Embedding Logical Queries on Knowledge Graphs
Learning low-dimensional embeddings of knowledge graphs is a powerful approach used to predict unobserved or missing edges between entities.
Ranked #6 on
Complex Query Answering
on FB15k-237
Automatic Detection of Incoherent Speech for Diagnosing Schizophrenia
Here, we present the first benchmark comparison of previously proposed coherence models for detecting symptoms of schizophrenia and evaluate their performance on a new dataset of recorded interviews between subjects and clinicians.
Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction
Translation-based methods for grammar correction that directly map noisy, ungrammatical text to their clean counterparts are able to correct a broad range of errors; however, such techniques are bottlenecked by the need for a large parallel corpus of noisy and clean sentence pairs.
Deconfounded Lexicon Induction for Interpretable Social Science
The first uses a bifurcated architecture to separate the explanatory power of the text and confounds.
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
We know very little about how neural language models (LM) use prior linguistic context.
Community Interaction and Conflict on the Web
Here we study intercommunity interactions across 36, 000 communities on Reddit, examining cases where users of one community are mobilized by negative sentiment to comment in another community.
Detecting Institutional Dialog Acts in Police Traffic Stops
We apply computational dialog methods to police body-worn camera footage to model conversations between police officers and community members in traffic stops.
Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes
Word embeddings use vectors to represent words such that the geometry between vectors captures semantic relationship between the words.
JESC: Japanese-English Subtitle Corpus
In this paper we describe the Japanese-English Subtitle Corpus (JESC).
Neural Net Models of Open-domain Discourse Coherence
In this paper, we describe domain-independent neural models of discourse coherence that are capable of measuring multiple aspects of coherence in existing sentences and can maintain coherence while generating new sentences.
Incorporating Dialectal Variability for Socially Equitable Language Identification
Language identification (LID) is a critical first step for processing multilingual text.
Community Identity and User Engagement in a Multi-Community Landscape
To this end we introduce a quantitative, language-based typology reflecting two key aspects of a community's identity: how distinctive, and how temporally dynamic it is.
A Two-stage Sieve Approach for Quote Attribution
We present a deterministic sieve-based system for attributing quotations in literary text and a new dataset: QuoteLi3.
Loyalty in Online Communities
In this paper we operationalize loyalty as a user-community relation: users loyal to a community consistently prefer it over all others; loyal communities retain their loyal users over time.
Data Noising as Smoothing in Neural Network Language Models
Data noising is an effective technique for regularizing neural network models.
Data Distillation for Controlling Specificity in Dialogue Generation
We show that from such a set of subsystems, one can use reinforcement learning to build a system that tailors its output to different input contexts at test time.
Adversarial Learning for Neural Dialogue Generation
In this paper, drawing intuition from the Turing test, we propose using adversarial training for open-domain dialogue generation: the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances.
Ranked #1 on
Dialogue Generation
on Amazon-5
Learning to Decode for Future Success
We introduce a simple, general strategy to manipulate the behavior of a neural decoder that enables it to generate outputs that have specific properties of interest (e. g., sequences of a pre-specified length).
Understanding Neural Networks through Representation Erasure
While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability.
A Simple, Fast Diverse Decoding Algorithm for Neural Generation
We further propose a variation that is capable of automatically adjusting its diversity decoding rates for different inputs using reinforcement learning (RL).
Citation Classification for Behavioral Analysis of a Scientific Field
Citations are an important indicator of the state of a scientific field, reflecting how authors frame their work, and influencing uptake by future scholars.
Cultural Shift or Linguistic Drift? Comparing Two Computational Measures of Semantic Change
Words shift in meaning for many reasons, including cultural factors like new technologies and regular linguistic processes like subjectification.
Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora
A word's sentiment depends on the domain in which it is used.
Deep Reinforcement Learning for Dialogue Generation
Recent neural models of dialogue generation offer great promise for generating responses for conversational agents, but tend to be shortsighted, predicting utterances one at a time while ignoring their influence on future outcomes.
Neural Net Models for Open-Domain Discourse Coherence
In this paper, we describe domain-independent neural models of discourse coherence that are capable of measuring multiple aspects of coherence in existing sentences and can maintain coherence while generating new sentences.
Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change
Understanding how words change their meanings over time is key to models of language and cultural evolution, but historical data on meaning is scarce, making theories hard to develop and test.
Neural Language Correction with Character-Based Attention
Motivated by these issues, we present a neural network-based approach to language correction.
Mutual Information and Diverse Decoding Improve Neural Machine Translation
We introduce an alternative objective function for neural MT that maximizes the mutual information between the source and target sentences, modeling the bi-directional dependency of sources and targets.
Learning multi-faceted representations of individuals from heterogeneous evidence using neural networks
Inferring latent attributes of people online is an important social computing task, but requires integrating the many heterogeneous sources of information available on the web.
Visualizing and Understanding Neural Models in NLP
While neural networks have been successfully applied to many NLP tasks the resulting vector-based models are very difficult to interpret.
Do Multi-Sense Embeddings Improve Natural Language Understanding?
Learning a distinct representation for each sense of an ambiguous word could lead to more powerful and fine-grained models of vector-space representations.
A Hierarchical Neural Autoencoder for Paragraphs and Documents
Natural language generation of coherent long texts like paragraphs or longer documents is a challenging problem for recurrent networks models.
When Are Tree Structures Necessary for Deep Learning of Representations?
Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture.
Inferring User Preferences by Probabilistic Logical Reasoning over Social Networks
by building a probabilistic model that reasons over user attributes (the user's location or gender) and the social network (the user's friends and spouse), via inferences like homophily (I am more likely to like sushi if spouse or friends like sushi, I am more likely to like the Knicks if I live in New York).
How to Ask for a Favor: A Case Study on the Success of Altruistic Requests
We present a case study of altruistic requests in an online community where all requests ask for the very same contribution and do not offer anything tangible in return, allowing us to disentangle what is requested from textual and social factors.
On the Importance of Text Analysis for Stock Price Prediction
We investigate the importance of text analysis for stock price prediction.
A Computational Approach to Politeness with Application to Social Factors
We propose a computational framework for identifying linguistic aspects of politeness.
