Search Results for author:
Found 30 papers, 14 papers with code
Rethinking Theory of Mind Benchmarks for LLMs: Towards A User-Centered Perspective
The last couple of years have witnessed emerging research that appropriates Theory-of-Mind (ToM) tasks designed for humans to benchmark LLM's ToM capabilities as an indication of LLM's social intelligence.
Interactive Agents to Overcome Ambiguity in Software Engineering
AI agents are increasingly being deployed to automate tasks, often based on ambiguous and underspecified user instructions.
AutoPresent: Designing Structured Visuals from Scratch
We benchmark end-to-end image generation and program generation methods with a variety of models, and find that programmatic methods produce higher-quality slides in user-interactable formats.
Bridging the Data Provenance Gap Across Text, Speech and Video
In this work we conduct the largest and first-of-its-kind longitudinal audit across modalities--popular text, speech, and video datasets--from their detailed sourcing trends and use restrictions to their geographical and linguistic representation.
TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet.
Minion: A Technology Probe for Resolving Value Conflicts through Expert-Driven and User-Driven Strategies in AI Companion Applications
Based on these, we created Minion, a technology probe to help users resolve human-AI value conflicts.
BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Data
In this work, we tackle the challenge of embedding realistic human personality traits into LLMs.
HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions
AI agents are increasingly autonomous in their interactions with human users and tools, leading to increased interactional safety risks.
AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents
We design a set of realistic scenarios where language agents are instructed to achieve goals that are in conflict with being truthful during a multi-turn conversation with simulated human agents.
On the Resilience of LLM-Based Multi-Agent Collaboration with Faulty Agents
Large language model-based multi-agent systems have shown great abilities across various tasks due to the collaboration of expert agents, each focusing on a specific domain.
Consent in Crisis: The Rapid Decline of the AI Data Commons
To our knowledge, we conduct the first, large-scale, longitudinal audit of the consent protocols for the web domains underlying AI training corpora.
PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models
Recent advances in large language models (LLMs) have led to their extensive global deployment, and ensuring their safety calls for comprehensive and multilingual toxicity evaluations.
Is this the real life? Is this just fantasy? The Misleading Success of Simulating Social Interactions With LLMs
Recent advances in large language models (LLM) have enabled richer social simulations, allowing for the study of various social phenomena.
Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.)
FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions
Theory of mind (ToM) evaluations currently focus on testing models using passive narratives that inherently lack interactivity.
SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents
We present SOTOPIA, an open-ended environment to simulate complex social interactions between artificial agents and evaluate their social intelligence.
WebArena: A Realistic Web Environment for Building Autonomous Agents
Building upon our environment, we release a set of benchmark tasks focusing on evaluating the functional correctness of task completions.
COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements
To study the contextual dynamics of offensiveness, we train models to generate COBRA explanations, with and without access to the context.
Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models
The escalating debate on AI's capabilities warrants developing reliable metrics to assess machine "intelligence".
Don't Take This Out of Context! On the Need for Contextual Models and Evaluations for Stylistic Rewriting
Most existing stylistic text rewriting methods and evaluation metrics operate on a sentence level, but ignoring the broader context of the text can lead to preferring generic, ambiguous, and incoherent rewrites.
Learning to translate by learning to communicate
We formulate and test a technique to use Emergent Communication (EC) with a pre-trained multilingual model to improve on modern Unsupervised NMT systems, especially for low-resource languages.
Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection
The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases.
Extracting and Inferring Personal Attributes from Dialogue
Personal attributes represent structured information about a person, such as their hobbies, pets, family, likes and dislikes.
Challenges in Automated Debiasing for Toxic Language Detection
Overall, our findings show that debiasing a model trained on biased toxic language data is not as effective as simply relabeling the data to remove existing biases.
Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets
Although large-scale pretrained language models, such as BERT and RoBERTa, have achieved superhuman performance on in-distribution test sets, their performance suffers on out-of-distribution test sets (e. g., on contrast sets).
Multilevel Text Alignment with Cross-Document Attention
Text alignment finds application in tasks such as citation recommendation and plagiarism detection.
RPD: A Distance Function Between Word Embeddings
Based on the properties of RPD, we study the relations of word embeddings of different algorithms systematically and investigate the influence of different training processes and corpora.
Evaluating Commonsense in Pre-trained Language Models
However, relatively little work has been done investigating commonsense knowledge contained in contextualized representations, which is crucial for human question answering and reading comprehension.
Parallel Distributed Logistic Regression for Vertical Federated Learning without Third-Party Coordinator
The system is built on the pa-rameter server architecture and aims to speed up the model training via utilizing a cluster of servers in case of large volume of training data.
Linking convolutional neural networks with graph convolutional networks: application in pulmonary artery-vein separation
In conclusion, the proposed CNN-GCN method combines local image information with graph connectivity information, improving pulmonary A/V separation over a baseline CNN method, approaching the performance of human observers.
Ranked #1 on
Pulmonary Artery–Vein Classification
on SunYs
3D Medical Imaging Segmentation
Pulmonary Artery–Vein Classification
