Search Results for author:
Found 46 papers, 4 papers with code
Human-Centered Evaluation of Explanations
The NLP community are increasingly interested in providing explanations for NLP models to help people make sense of model behavior and potentially improve human interaction with models.
Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies
Large language models (LLMs) can produce erroneous responses that sound fluent and convincing, raising the risk that users will rely on these responses as if they were correct.
As Confidence Aligns: Exploring the Effect of AI Confidence on Human Self-confidence in Human-AI Decision Making
Complementary collaboration between humans and AI is essential for human-AI decision making.
"It was 80% me, 20% AI": Seeking Authenticity in Co-Writing with Large Language Models
Given the rising proliferation and diversity of AI writing assistance tools, especially those powered by large language models (LLMs), both writers and readers may have concerns about the impact of these tools on the authenticity of writing work.
ECBD: Evidence-Centered Benchmark Design for NLP
Benchmarking is seen as critical to assessing progress in NLP.
"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trust
However, there has been little empirical work examining how users perceive and act upon LLMs' expressions of uncertainty.
Generative Echo Chamber? Effects of LLM-Powered Search Systems on Diverse Information Seeking
Large language models (LLMs) powered conversational search systems have already been used by hundreds of millions of people, and are believed to bring many benefits over conventional search.
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap
It is paramount to pursue new approaches to provide transparency for LLMs, and years of research at the intersection of AI and human-computer interaction (HCI) highlight that we must do so with a human-centered perspective: Transparency is fundamentally about supporting appropriate human understanding, and this understanding is sought by different stakeholders with different goals in different contexts.
Rethinking Model Evaluation as Narrowing the Socio-Technical Gap
The recent development of generative large language models (LLMs) poses new challenges for model evaluation that the research community and industry have been grappling with.
Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theory
We address a fundamental challenge in Natural Language Generation (NLG) model evaluation -- the design and evaluation of evaluation metrics.
Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes.
Why is AI not a Panacea for Data Workers? An Interview Study on Human-AI Collaboration in Data Storytelling
Data storytelling plays an important role in data workers' daily jobs since it boosts team collaboration and public communication.
fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Tasks
To design with AI models, user experience (UX) designers must assess the fit between the model and user needs.
Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
To address this problem, we bridge the literature on AI design and AI transparency to explore whether and how frameworks for transparent model reporting can support design ideation with pre-trained models.
Human-Centered Responsible Artificial Intelligence: Current & Future Trends
In recent years, the CHI community has seen significant growth in research on Human-Centered Responsible Artificial Intelligence.
Generation Probabilities Are Not Enough: Uncertainty Highlighting in AI Code Completions
Through a mixed-methods study with 30 programmers, we compare three conditions: providing the AI system's code completion alone, highlighting tokens with the lowest likelihood of being generated by the underlying generative model, and highlighting tokens with the highest predicted likelihood of being edited by a programmer.
Selective Explanations: Leveraging Human Input to Align Explainable AI
As a result, current XAI techniques are often found to be hard to use and lack effectiveness.
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
AI explanations are often mentioned as a way to improve human-AI decision-making, but empirical studies have not found consistent evidence of explanations' effectiveness and, on the contrary, suggest that they can increase overreliance when the AI system is wrong.
Seamful XAI: Operationalizing Seamful Design in Explainable AI
We found that the Seamful XAI design process helped users foresee AI harms, identify underlying reasons (seams), locate them in the AI's lifecycle, learn how to leverage seamful information to improve XAI and user agency.
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
We argue that one way to close the gap is to develop evaluation methods that account for different user requirements in these usage contexts.
Designing for Responsible Trust in AI Systems: A Communication Perspective
Current literature and public discourse on "trust in AI" are often focused on the principles underlying trustworthy AI, with insufficient attention paid to how people develop trust.
Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation
Despite impressive performance in many benchmark datasets, AI models can still make mistakes, especially among out-of-distribution examples.
Investigating Explainability of Generative AI for Code through Scenario-based Design
Using scenario-based design and question-driven XAI design approaches, we explore users' explainability needs for GenAI in three software engineering use cases: natural language to code, code translation, and code auto-completion.
Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies
Besides developing AI technologies for this purpose, the emerging field of human-AI decision making must embrace empirical approaches to form a foundational understanding of how humans interact and work with AI to make decisions.
Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
In this chapter, we begin with a high-level overview of the technical landscape of XAI algorithms, then selectively survey our own and other recent HCI works that take human-centered approaches to design, evaluate, and provide conceptual and methodological tools for XAI.
AI Explainability 360: Impact and Design
As artificial intelligence and machine learning algorithms become increasingly prevalent in society, multiple stakeholders are calling for these algorithms to provide explanations.
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Explainability of AI systems is critical for users to take informed actions.
Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI
In this paper, we describe an open source Python toolkit named Uncertainty Quantification 360 (UQ360) for the uncertainty quantification of AI models.
Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
Automated Machine Learning (AutoML) is a rapidly growing set of technologies that automate the model development pipeline by searching model space and generating candidate models.
Question-Driven Design Process for Explainable AI User Experiences
A pervasive design issue of AI systems is their explainability--how to provide appropriate information to help users understand the AI.
Facilitating Knowledge Sharing from Domain Experts to Data Scientists for Building NLP Models
Data scientists face a steep learning curve in understanding a new domain for which they want to build machine learning (ML) models.
Expanding Explainability: Towards Social Transparency in AI systems
We suggested constitutive design elements of ST and developed a conceptual framework to unpack ST's effect and implications at the technical, decision-making, and organizational level.
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Explainability attempts to provide reasons for a machine learning model's behavior to stakeholders.
Agent Assist through Conversation Analysis
Customer support agents play a crucial role as an interface between an organization and its end-users.
Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation
The similarity score between feature rankings provided by the annotator and the local model explanation is used to assign a weight to each corresponding committee model.
Measuring Social Biases of Crowd Workers using Counterfactual Queries
Social biases based on gender, race, etc.
Explainable Active Learning (XAL): An Empirical Study of How Local Explanations Impact Annotator Experience
We conducted an empirical study comparing the model learning outcomes, feedback content and experience with XAL, to that of traditional AL and coactive learning (providing the model's prediction without the explanation).
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
A surge of interest in explainable AI (XAI) has led to a vast collection of algorithmic work on the topic.
Explainable Artificial Intelligence (XAI)
Open-Ended Question Answering
Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Making
In these scenarios, full automation is often undesirable, not only due to the significance of the outcome, but also because human experts can draw on their domain knowledge complementary to the model's to ensure task success.
Enabling Value Sensitive AI Systems through Participatory Design Fictions
Two general routes have been followed to develop artificial agents that are sensitive to human values---a top-down approach to encode values into the agents, and a bottom-up approach to learn from human actions, whether from real-world interactions or stories.
Doc2Dial: a Framework for Dialogue Composition Grounded in Business Documents
We introduce Doc2Dial, an end-to-end framework for generating conversational data grounded in business documents via crowdsourcing.
One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
Equally important, we provide a taxonomy to help entities requiring explanations to navigate the space of explanation methods, not only those in the toolkit but also in the broader literature on explainability.
Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions
The rise of increasingly more powerful chatbots offers a new way to collect information through conversational surveys, where a chatbot asks open-ended questions, interprets a user's free-text responses, and probes answers whenever needed.
Bootstrapping Conversational Agents With Weak Supervision
We report on a user study that shows positive user feedback for this new approach to build conversational agents, and demonstrates the effectiveness of using data programming for auto-labeling.
A Measure for Dialog Complexity and its Application in Streamlining Service Operations
Dialog is a natural modality for interaction between customers and businesses in the service industry.
