1 code implementation • 21 Jun 2019 • Manish Raghavan, Solon Barocas, Jon Kleinberg, Karen Levy
How are algorithmic assessments built, validated, and examined for bias?
1 code implementation • 28 May 2020 • Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
We survey 146 papers analyzing "bias" in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing "bias" is an inherently normative process.
no code implementations • ACL 2020 • Su Lin Blodgett, Solon Barocas, Hal Daum{\'e} III, Hanna Wallach
We survey 146 papers analyzing {``}bias{''} in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing {``}bias{''} is an inherently normative process.
no code implementations • 10 Mar 2021 • Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple.
no code implementations • 26 May 2021 • Margot Hanley, Solon Barocas, Karen Levy, Shiri Azenkot, Helen Nissenbaum
In this paper, we investigate the ethical dilemmas faced by companies that have adopted the use of computer vision for producing alt text: textual descriptions of images for blind and low vision people, We use Facebook's automatic alt text tool as our primary case study.
no code implementations • 8 Mar 2022 • Fernando Delgado, Solon Barocas, Karen Levy
Despite growing calls for participation in AI design, there are to date few empirical studies of what these processes look like and how they can be structured for meaningful engagement with domain experts.
1 code implementation • 5 May 2022 • Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
Transparency around limitations can improve the scientific rigor of research, help ensure appropriate interpretation of research findings, and make research claims more credible.
no code implementations • 14 Jun 2022 • Angelina Wang, Solon Barocas, Kristen Laird, Hanna Wallach
We propose multiple measurement techniques for each type of harm.
no code implementations • 19 Jul 2022 • Reid McIlroy-Young, Jon Kleinberg, Siddhartha Sen, Solon Barocas, Ashton Anderson
An emerging theme in artificial intelligence research is the creation of models to simulate the decisions and behavior of specific people, in domains including game-playing, text generation, and artistic expression.
1 code implementation • 27 Jan 2023 • A. Feder Cooper, Katherine Lee, Madiha Zahrah Choksi, Solon Barocas, Christopher De Sa, James Grimmelmann, Jon Kleinberg, Siddhartha Sen, Baobao Zhang
Variance in predictions across different trained models is a significant, under-explored source of error in fair binary classification.
no code implementations • 28 Jan 2023 • Hoda Heidari, Solon Barocas, Jon Kleinberg, Karen Levy
Prior work has provided strong evidence that, within organizational settings, teams that bring a diversity of information and perspectives to a task are more effective than teams that do not.
1 code implementation • 23 Jun 2023 • Jamelle Watson-Daniels, Solon Barocas, Jake M. Hofman, Alexandra Chouldechova
Along the way, we refine the study of single-target multiplicity by introducing notions of multiplicity that respect resource constraints -- a feature of many real-world tasks that is not captured by existing notions of predictive multiplicity.
no code implementations • 8 Sep 2023 • Lydia T. Liu, Solon Barocas, Jon Kleinberg, Karen Levy
Through a simple model encompassing actions, latent states, and measurements, we demonstrate that pure outcome prediction rarely results in the most effective policy for taking actions, even when combined with other measurements.
no code implementations • 6 Feb 2024 • Angelina Wang, Xuechunzi Bai, Solon Barocas, Su Lin Blodgett
However, certain stereotype-violating errors are more experientially harmful for men, potentially due to perceived threats to masculinity.