Search Results for author:
Found 26 papers, 10 papers with code
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice
We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy.
A Framework for Evaluating LLMs Under Task Indeterminacy
Large language model (LLM) evaluations often assume there is a single correct response -- a gold label -- for each item in the evaluation corpus.
SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluation
Today, it is common for multiple clients to procure the same AI model from a model developer, and the task of disaggregated evaluation is faced by each customer individually.
A structured regression approach for evaluating model performance across intersectional subgroups
Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups defined by combinations of demographic or other sensitive attributes.
The Impact of Differential Feature Under-reporting on Algorithmic Fairness
Predictive risk models in the public sector are commonly developed using administrative data that is more complete for subpopulations that more greatly rely on public services.
Multi-Target Multiplicity: Flexibility and Fairness in Target Specification under Resource Constraints
Along the way, we refine the study of single-target multiplicity by introducing notions of multiplicity that respect resource constraints -- a feature of many real-world tasks that is not captured by existing notions of predictive multiplicity.
Examining risks of racial biases in NLP tools for child protective services
Given well-established racial bias in this setting, we investigate possible ways deployed NLP is liable to increase racial disparities.
Algorithmic Fairness and Vertical Equity: Income Fairness with IRS Tax Audit Models
First, we show how the use of more flexible machine learning (classification) methods -- as opposed to simpler models -- shifts audit burdens from high to middle-income taxpayers.
Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholders
In this work, we conducted a set of seven design workshops with 35 stakeholders who have been impacted by the child welfare system or who work in it to understand their beliefs and concerns around PRMs, and to engage them in imagining new uses of data and technologies in the child welfare system.
Doubting AI Predictions: Influence-Driven Second Opinion Recommendation
Effective human-AI collaboration requires a system design that provides humans with meaningful ways to make sense of and critically evaluate algorithmic recommendations.
Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness
In many settings, however, the final prediction or decision of a system is under the control of a human, who uses an algorithm's output along with their own personal expertise in order to produce a combined prediction.
The Impact of Algorithmic Risk Assessments on Human Predictions and its Analysis via Crowdsourcing Studies
As algorithmic risk assessment instruments (RAIs) are increasingly adopted to assist decision makers, their predictive performance and potential to promote inequity have come under scrutiny.
The effect of differential victim crime reporting on predictive policing systems
Our analysis is based on a simulation patterned after district-level victimization and crime reporting survey data for Bogot\'a, Colombia.
Leveraging Expert Consistency to Improve Algorithmic Decision Support
However, there is frequently a construct gap: a gap between the construct of interest to the decision-making task and what is captured in proxies used as labels to train ML models.
Characterizing Fairness Over the Set of Good Models Under Selective Labels
We develop a framework for characterizing predictive fairness properties over the set of models that deliver similar overall performance, or "the set of good models."
Counterfactual Predictions under Runtime Confounding
We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
Counterfactual Risk Assessments, Evaluation, and Fairness
These tools thus reflect risk under the historical policy, rather than under the different decision options that the tool is intended to inform.
What's in a Name? Reducing Bias in Bios without Access to Protected Attributes
In the context of mitigating bias in occupation classification, we propose a method for discouraging correlation between the predicted probability of an individual's true occupation and a word embedding of their name.
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives.
The Frontiers of Fairness in Machine Learning
The last few years have seen an explosion of academic and popular interest in algorithmic fairness.
Learning under selective labels in the presence of expert consistency
We explore the problem of learning under selective labels in the context of algorithm-assisted decision making.
Does mitigating ML's impact disparity require treatment disparity?
Following related work in law and policy, two notions of disparity have come to shape the study of fairness in algorithmic decision-making.
Fairer and more accurate, but for whom?
Complex statistical machine learning models are increasingly being used or considered for use in high-stakes decision-making pipelines in domains such as financial services, health care, criminal justice and human services.
Fair prediction with disparate impact: A study of bias in recidivism prediction instruments
Recidivism prediction instruments (RPI's) provide decision makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time.
Fair prediction with disparate impact: A study of bias in recidivism prediction instruments
Recidivism prediction instruments provide decision makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time.
Generalized Additive Model Selection
We introduce GAMSEL (Generalized Additive Model Selection), a penalized likelihood approach for fitting sparse generalized additive models in high dimension.
