Search Results for author:
Found 15 papers, 4 papers with code
The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmers
Evaluation of large language models (LLMs) for code has primarily relied on static benchmarks, including HumanEval (Chen et al., 2021), which measure the ability of LLMs to generate complete code that passes unit tests.
Do LLMs exhibit human-like response biases? A case study in survey design
As large language models (LLMs) become more capable, there is growing excitement about the possibility of using LLMs as proxies for humans in real-world tasks where subjective labels are desired, such as in surveys and opinion polling.
AdvisingNets: Learning to Distinguish Correct and Wrong Classifications via Nearest-Neighbor Explanations
Besides providing insights into how an image classifier makes its predictions, nearest-neighbor examples also help humans make more accurate decisions.
FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines
We propose FeedbackLogs, addenda to existing documentation of ML pipelines, to track the input of multiple stakeholders.
Learning Personalized Decision Support Policies
In this work, we propose learning a decision support policy that, for a given input, chooses which form of support, if any, to provide.
Assisting Human Decisions in Document Matching
Many practical applications, ranging from paper-reviewer assignment in peer review to job-applicant matching for hiring, require human decision makers to identify relevant matches by combining their expertise with predictions from machine learning models.
A Case Study on Designing Evaluations of ML Explanations with Simulated User Studies
We hope that this work motivates further study of when and how SimEvals should be used to aid in the design of real-world evaluations.
Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations
AI explanations are often mentioned as a way to improve human-AI decision-making, but empirical studies have not found consistent evidence of explanations' effectiveness and, on the contrary, suggest that they can increase overreliance when the AI system is wrong.
On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods
Most existing evaluations of explainable machine learning (ML) methods rely on simplifying assumptions or proxies that do not reflect real-world use cases; the handful of more robust evaluations on real-world settings have shortcomings in their design, resulting in limited conclusions of methods' real-world utility.
Use-Case-Grounded Simulations for Explanation Evaluation
SimEvals involve training algorithmic agents that take as input the information content (such as model explanations) that would be presented to each participant in a human subject study, to predict answers to the use case of interest.
Perspectives on Incorporating Expert Feedback into Model Updates
A practitioner may receive feedback from an expert at the observation- or domain-level, and convert this feedback into updates to the dataset, loss function, or parameter space.
Bayesian Persuasion for Algorithmic Recourse
While the decision maker's problem of finding the optimal Bayesian incentive-compatible (BIC) signaling policy takes the form of optimization over infinitely-many variables, we show that this optimization can be cast as a linear program over finitely-many regions of the space of possible assessment rules.
Interpretable Machine Learning: Moving From Mythos to Diagnostics
Despite increasing interest in the field of Interpretable Machine Learning (IML), a significant gap persists between the technical objectives targeted by researchers' methods and the high-level goals of consumers' use cases.
Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning
We also find that incorporating natural language allows the model to generalize to unseen tasks in a zero-shot setting and to learn quickly from a few demonstrations.
Novelty Detection via Network Saliency in Visual-based Deep Learning
Machine-learning driven safety-critical autonomous systems, such as self-driving cars, must be able to detect situations where its trained model is not able to make a trustworthy prediction.
