Search Results for author:
Found 93 papers, 44 papers with code
Uncovering Surprising Event Boundaries in Narratives
It is important to define meaningful and interpretable automatic evaluation metrics for open-domain dialog research.
Misinfo Reaction Frames: Reasoning about Readers’ Reactions to News Headlines
Even to a simple and short news headline, readers react in a multitude of ways: cognitively (e. g. inferring the writer’s intent), emotionally (e. g. feeling distrust), and behaviorally (e. g. sharing the news with their friends).
Crowdsourcing Beyond Annotation: Case Studies in Benchmark Data Collection
Even though it is such a fundamental tool in NLP, crowdsourcing use is largely guided by common practices and the personal experience of researchers.
Exploring Big Five Personality and AI Capability Effects in LLM-Simulated Negotiation Dialogues
This paper presents an evaluation framework for agentic AI systems in mission-critical negotiation contexts, addressing the need for AI agents that can adapt to diverse human operators and stakeholders.
Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferences
Current LLMs are trained to refuse potentially harmful input queries regardless of whether users actually had harmful intents, causing a tradeoff between safety and user experience.
Breaking mBad! Supervised Fine-tuning for Cross-Lingual Detoxification
As large language models (LLMs) become increasingly prevalent in global applications, ensuring that they are toxicity-free across diverse linguistic contexts remains a critical challenge.
SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulation
Social simulation through large language model (LLM) agents is a promising approach to explore and validate hypotheses related to social science questions and LLM agents behavior.
Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective
The last couple of years have witnessed emerging research that appropriates Theory-of-Mind (ToM) tasks designed for humans to benchmark LLM's ToM capabilities as an indication of LLM's social intelligence.
Out of Style: RAG's Fragility to Linguistic Variation
Despite the impressive performance of Retrieval-augmented Generation (RAG) systems across various NLP benchmarks, their robustness in handling real-world user-LLM interaction queries remains largely underexplored.
Un-Straightening Generative AI: How Queer Artists Surface and Challenge the Normativity of Generative AI Models
Queer people are often discussed as targets of bias, harm, or discrimination in research on generative AI.
Medical Hallucinations in Foundation Models and Their Impact on Healthcare
Our contributions include (1) a taxonomy for understanding and addressing medical hallucinations, (2) benchmarking models using medical hallucination dataset and physician-annotated LLM responses to real medical cases, providing direct insight into the clinical impact of hallucinations, and (3) a multi-national clinician survey on their experiences with medical hallucinations.
Mind the Gesture: Evaluating AI Sensitivity to Culturally Offensive Non-Verbal Gestures
Gestures are an integral part of non-verbal communication, with meanings that vary across cultures, and misinterpretations that can have serious social and diplomatic consequences.
Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning
Large language models (LLMs) often fail to ask effective questions under uncertainty, making them unreliable in domains where proactive information-gathering is essential for decisionmaking.
Interactive Agents to Overcome Ambiguity in Software Engineering
AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.
Rejected Dialects: Biases Against African American Language in Reward Models
Preference alignment via reward models helps build safe, helpful, and reliable large language models (LLMs).
AutoPresent: Designing Structured Visuals from Scratch
We benchmark end-to-end image generation and program generation methods with a variety of models, and find that programmatic methods produce higher-quality slides in user-interactable formats.
Multi-Attribute Constraint Satisfaction via Language Model Rewriting
Thus, we create Multi-Attribute Constraint Satisfaction (MACS), a generalized method capable of finetuning language models on any sequential domain to satisfy user-specified constraints on multiple external real-value attributes.
Minion: A Technology Probe for Resolving Value Conflicts through Expert-Driven and User-Driven Strategies in AI Companion Applications
Based on these, we created Minion, a technology probe to help users resolve human-AI value conflicts.
SafetyAnalyst: Interpretable, Transparent, and Steerable Safety Moderation for AI Behavior
The ideal AI safety moderation system would be both structurally interpretable (so its decisions can be reliably explained) and steerable (to align to safety standards and reflect a community's values), which current systems fall short on.
BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data
In this work, we tackle the challenge of embedding realistic human personality traits into LLMs.
Data Defenses Against Large Language Models
Large language models excel at performing inference over text to extract information, summarize information, or generate additional text.
HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions
AI agents are increasingly autonomous in their interactions with human users and tools, leading to increased interactional safety risks.
AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents
We design a set of realistic scenarios where language agents are instructed to achieve goals that are in conflict with being truthful during a multi-turn conversation with simulated human agents.
On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents
Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents, each focusing on a specific domain.
Rel-A.I.: An Interaction-Centered Approach To Measuring Human-LM Reliance
The ability to communicate uncertainty, risk, and limitation is crucial for the safety of large language models.
WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
As WildJailbreak considerably upgrades the quality and scale of existing safety resources, it uniquely enables us to examine the scaling effects of data and the interplay of data properties and model capabilities during safety training.
HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMs
We introduce a novel, theory-based taxonomy, HEART (Human Empathy and Narrative Taxonomy) that delineates elements of narrative style that can lead to empathy with the narrator of a story.
PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models
Recent advances in large language models (LLMs) have led to their extensive global deployment, and ensuring their safety calls for comprehensive and multilingual toxicity evaluations.
Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Non-Literal Intent Resolution in LLMs
Humans often express their communicative intents indirectly or non-literally, which requires their interlocutors -- human or AI -- to understand beyond the literal meaning of words.
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
We introduce NormAd, an evaluation framework to assess LLMs' cultural adaptability, specifically measuring their ability to judge social acceptability across different levels of cultural norm specificity, from abstract values to explicit social norms.
Particip-AI: A Democratic Surveying Framework for Anticipating Future AI Use Cases, Harms and Benefits
General purpose AI, such as ChatGPT, seems to have lowered the barriers for the public to use AI and harness its power.
SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents
Motivated by this gap, we propose an interactive learning method, SOTOPIA-$\pi$, improving the social intelligence of language agents.
Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena.
Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty
As natural language becomes the default interface for human-AI interaction, there is a need for LMs to appropriately communicate uncertainties in downstream applications.
Riveter: Measuring Power and Social Dynamics Between Entities
Riveter provides a complete easy-to-use pipeline for analyzing verb connotations associated with entities in text corpora.
Where Do People Tell Stories Online? Story Detection Across Online Communities
Story detection in online communities is a challenging task as stories are scattered across communities and interwoven with non-storytelling spans within a single text.
Beyond Denouncing Hate: Strategies for Countering Implied Biases and Stereotypes in Language
Counterspeech, i. e., responses to counteract potential harms of hateful speech, has become an increasingly popular solution to address online hate speech without censorship.
Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.)
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity.
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence.
Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Duties
To improve AI systems to better reflect value pluralism, the first-order challenge is to explore the extent to which AI systems can model pluralistic human values, rights, and duties as well as their interaction.
COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements
To study the contextual dynamics of offensiveness, we train models to generate COBRA explanations, with and without access to the context.
NLPositionality: Characterizing Design Biases of Datasets and Models
We introduce NLPositionality, a framework for characterizing design biases and quantifying the positionality of NLP datasets and models.
From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models
Dogwhistles are coded expressions that simultaneously convey one meaning to a broad audience and a second one, often hateful or provocative, to a narrow in-group; they are deployed to evade both political repercussions and algorithmic content moderation.
Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
However, RLHF is an unstable and data-hungry process that continually requires new high-quality LM-generated data for finetuning.
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
The escalating debate on AI's capabilities warrants developing reliable metrics to assess machine "intelligence".
Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting
Most existing stylistic text rewriting methods and evaluation metrics operate on a sentence level, but ignoring the broader context of the text can lead to preferring generic, ambiguous, and incoherent rewrites.
BiasX: "Thinking Slow" in Toxic Content Moderation with Explanations of Implied Social Biases
Toxicity annotators and content moderators often default to mental shortcuts when making decisions.
Modeling Empathic Similarity in Personal Narratives
The most meaningful connections between people are often fostered through expression of shared vulnerability and emotional experiences in personal narratives.
Queer In AI: A Case Study in Community-Led Participatory AI
We present Queer in AI as a case study for community-led participatory design in AI.
Towards Countering Essentialism through Social Bias Reasoning
Essentialist beliefs (i. e., believing that members of the same group are fundamentally alike) play a central role in social stereotypes and can lead to harm when left unchallenged.
Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts
Text detoxification has the potential to mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle.
SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization
Data scarcity has been a long standing issue in the field of open-domain social dialogue.
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
We show that one of today's largest language models (GPT-3; Brown et al., 2020) lacks this kind of social intelligence out-of-the box, using two tasks: SocialIQa (Sap et al., 2019), which measures models' ability to understand intents and reactions of participants of social interactions, and ToMi (Le et al., 2019), which measures whether models can infer mental states and realities of participants of situations.
When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment
Using a state-of-the-art large language model (LLM) as a basis, we propose a novel moral chain of thought (MORALCOT) prompting strategy that combines the strengths of LLMs with theories of moral reasoning developed in cognitive science to predict human moral judgments.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
BIG-bench focuses on tasks that are believed to be beyond the capabilities of current language models.
ProsocialDialog: A Prosocial Backbone for Conversational Agents
With this dataset, we introduce a dialogue safety detection module, Canary, capable of generating RoTs given conversational context, and a socially-informed dialogue agent, Prost.
Ranked #1 on
Dialogue Safety Prediction
on ProsocialDialog
Aligning to Social Norms and Values in Interactive Narratives
We focus on creating agents that act in alignment with socially beneficial norms and values in interactive narratives or text-based games -- environments wherein an agent perceives and interacts with a world through natural language.
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection
To help mitigate these issues, we create ToxiGen, a new large-scale and machine-generated dataset of 274k toxic and benign statements about 13 minority groups.
Imagined versus Remembered Stories: Quantifying Differences in Narrative Flow
We quantify the differences between autobiographical and imagined stories by introducing sequentiality, a measure of narrative flow of events, drawing probabilistic inferences from a cutting-edge large language model (GPT-3).
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases.
Can Machines Learn Morality? The Delphi Experiment
As AI systems become increasingly powerful and pervasive, there are growing concerns about machines' morality or a lack thereof.
Just Say No: Analyzing the Stance of Neural Dialogue Generation in Offensive Contexts
To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations.
DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts
Despite recent advances in natural language generation, it remains challenging to control attributes of generated text.
Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlines
We propose Misinfo Reaction Frames (MRF), a pragmatic formalism for modeling how readers might react to a news headline.
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus
Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.
Detoxifying Language Models Risks Marginalizing Minority Voices
Language models (LMs) must be both safe and equitable to be responsibly deployed in practice.
Challenges in Automated Debiasing for Toxic Language Detection
Overall, our findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases.
Social Chemistry 101: Learning to Reason about Social and Moral Norms
We present Social Chemistry, a new conceptual formalism to study people's everyday social norms and moral judgments over a rich spectrum of real life situations described in natural language.
PowerTransformer: Unsupervised Controllable Revision for Biased Language Correction
Unconscious biases continue to be prevalent in modern text and media, calling for algorithms that can assist writers with bias correction.
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
Recollection versus Imagination: Exploring Human Memory and Cognition via Neural Language Models
We introduce a measure of narrative flow and use this to examine the narratives for imagined and recalled events.
Exploring the Effect of Author and Reader Identity in Online Story Writing: the STORIESINTHEWILD Corpus.
We analyze the effect of author and reader characteristics and story writing setup on the quality of stories in a short storytelling task.
Commonsense Reasoning for Natural Language Processing
We organize this tutorial to provide researchers with the critical foundations and recent advances in commonsense representation and reasoning, in the hopes of casting a brighter light on this promising area of future research.
Social Bias Frames: Reasoning about Social and Power Implications of Language
We introduce Social Bias Frames, a new conceptual formalism that aims to model the pragmatic frames in which people project social biases and stereotypes onto others.
Social IQa: Commonsense Reasoning about Social Interactions
We introduce Social IQa, the first large-scale benchmark for commonsense reasoning about social situations.
The Risk of Racial Bias in Hate Speech Detection
We investigate how annotators{'} insensitivity to differences in dialect can lead to racial bias in automatic hate speech detection models, potentially amplifying harm against minority populations.
COMET: Commonsense Transformers for Automatic Knowledge Graph Construction
We present the first comprehensive study on automatic knowledge base construction for two prevalent commonsense knowledge graphs: ATOMIC (Sap et al., 2019) and ConceptNet (Speer et al., 2017).
SocialIQA: Commonsense Reasoning about Social Interactions
We introduce Social IQa, the first largescale benchmark for commonsense reasoning about social situations.
Ranked #13 on
Question Answering
on SIQA
ATOMIC: An Atlas of Machine Commonsense for If-Then Reasoning
We present ATOMIC, an atlas of everyday commonsense reasoning, organized through 877k textual descriptions of inferential knowledge.
Event2Mind: Commonsense Inference on Events, Intents, and Reactions
We investigate a new commonsense inference task: given an event described in a short free-form text ("X drinks coffee in the morning"), a system reasons about the likely intents ("X wants to stay awake") and reactions ("X feels alert") of the event's participants.
Ranked #1 on
Common Sense Reasoning
on Event2Mind test
Modeling Naive Psychology of Characters in Simple Commonsense Stories
Understanding a narrative requires reading between the lines and reasoning about the unspoken but obvious implications about events and people's mental states - a capability that is trivial for humans but remarkably hard for machines.
Ranked #2 on
Emotion Classification
on ROCStories
Sounding Board: A User-Centric and Content-Driven Social Chatbot
We present Sounding Board, a social chatbot that won the 2017 Amazon Alexa Prize.
DLATK: Differential Language Analysis ToolKit
We present Differential Language Analysis Toolkit (DLATK), an open-source python package and command-line tool developed for conducting social-scientific language analyses.
Connotation Frames of Power and Agency in Modern Films
The framing of an action influences how we perceive its actor.
Story Cloze Task: UW NLP System
This paper describes University of Washington NLP{'}s submission for the Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem 2017) shared task{---}the Story Cloze Task.
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
A writer's style depends not just on personal traits but also on her intent and mental state.
