no code implementations • 1 Apr 2025 • Emily Corvi, Hannah Washington, Stefanie Reed, Chad Atalla, Alexandra Chouldechova, P. Alex Dow, Jean Garcia-Gathright, Nicholas Pangakis, Emily Sheng, Dan Vann, Matthew Vogel, Hanna Wallach
We present a framework, grounded in speech act theory (Austin, 1962), that conceptualizes representational harms caused by generative language systems as the perlocutionary effects (i. e., real-world impacts) of particular types of illocutionary acts (i. e., system behaviors).
no code implementations • 7 Mar 2025 • Luke Guerdan, Solon Barocas, Kenneth Holstein, Hanna Wallach, Zhiwei Steven Wu, Alexandra Chouldechova
High agreement rates between these gold labels and judge system ratings are then taken as a sign of good judge system performance.
no code implementations • 7 Mar 2025 • Laura Weidinger, Inioluwa Deborah Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Deep Ganguli, Sanmi Koyejo, William Isaac
There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts.
no code implementations • 9 Dec 2024 • A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, Amy Cyphert, Mark Lemley, danah boyd, Jennifer Wortman Vaughan, Miles Brundage, David Bau, Seth Neel, Abigail Z. Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy.
no code implementations • 21 Nov 2024 • Luke Guerdan, Hanna Wallach, Solon Barocas, Alexandra Chouldechova
Large language model (LLM) evaluations often assume there is a single correct response -- a gold label -- for each item in the evaluation corpus.
no code implementations • 26 Oct 2023 • Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen
We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services.
no code implementations • 23 Oct 2023 • Li Lucy, Su Lin Blodgett, Milad Shokouhi, Hanna Wallach, Alexandra Olteanu
Fairness-related assumptions about what constitute appropriate NLG system behaviors range from invariance, where systems are expected to behave identically for social groups, to adaptation, where behaviors should instead vary across them.
1 code implementation • Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics 2023 • Eve Fleisig, Aubrie Amstutz, Chad Atalla, Su Lin Blodgett, Hal Daumé III, Alexandra Olteanu, Emily Sheng, Dan Vann, Hanna Wallach
It is critical to measure and mitigate fairness- related harms caused by AI text generation systems, including stereotyping and demeaning harms.
no code implementations • 14 Jun 2022 • Angelina Wang, Solon Barocas, Kristen Laird, Hanna Wallach
We propose multiple measurement techniques for each type of harm.
no code implementations • 6 Jun 2022 • Amy K. Heger, Liz B. Marquis, Mihaela Vorvoreanu, Hanna Wallach, Jennifer Wortman Vaughan
Despite the fact that data documentation frameworks are often motivated from the perspective of responsible AI, participants did not make the connection between the questions that they were asked to answer and their responsible AI implications.
1 code implementation • 5 May 2022 • Jessie J. Smith, Saleema Amershi, Solon Barocas, Hanna Wallach, Jennifer Wortman Vaughan
Transparency around limitations can improve the scientific rigor of research, help ensure appropriate interpretation of research findings, and make research claims more credible.
no code implementations • 10 Dec 2021 • Michael Madaio, Lisa Egede, Hariharan Subramonyam, Jennifer Wortman Vaughan, Hanna Wallach
Various tools and practices have been developed to support practitioners in identifying, assessing, and mitigating fairness-related harms caused by AI systems.
no code implementations • ACL 2021 • Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna Wallach
Auditing NLP systems for computational harms like surfacing stereotypes is an elusive goal.
no code implementations • 12 Jun 2021 • Aaron Schein, Anjali Nagulpally, Hanna Wallach, Patrick Flaherty
We present a new non-negative matrix factorization model for $(0, 1)$ bounded-support data based on the doubly non-central beta (DNCB) distribution, a generalization of the beta distribution.
1 code implementation • 27 Apr 2021 • David Alvarez-Melis, Harmanpreet Kaur, Hal Daumé III, Hanna Wallach, Jennifer Wortman Vaughan
We take inspiration from the study of human explanation to inform the design and evaluation of interpretability methods in machine learning.
BIG-bench Machine Learning
Interpretable Machine Learning
+1
no code implementations • 10 Mar 2021 • Solon Barocas, Anhong Guo, Ece Kamar, Jacquelyn Krones, Meredith Ringel Morris, Jennifer Wortman Vaughan, Duncan Wadsworth, Hanna Wallach
Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple.
no code implementations • ACL 2020 • Su Lin Blodgett, Solon Barocas, Hal Daum{\'e} III, Hanna Wallach
We survey 146 papers analyzing {``}bias{''} in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing {``}bias{''} is an inherently normative process.
1 code implementation • 28 May 2020 • Su Lin Blodgett, Solon Barocas, Hal Daumé III, Hanna Wallach
We survey 146 papers analyzing "bias" in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing "bias" is an inherently normative process.
no code implementations • 3 May 2020 • Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damián Blasi, Hanna Wallach
We also find that there are statistically significant relationships between the grammatical genders of inanimate nouns and the verbs that take those nouns as direct objects, as indirect objects, and as subjects.
no code implementations • 11 Dec 2019 • Abigail Z. Jacobs, Hanna Wallach
We argue that this contestedness underlies recent debates about fairness definitions: although these debates appear to be about different operationalizations, they are, in fact, debates about different theoretical understandings of fairness.
no code implementations • IJCNLP 2019 • Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin, Damián Blasi, Hanna Wallach
To that end, we use canonical correlation analysis to correlate the grammatical gender of inanimate nouns with an externally grounded definition of their lexical semantics.
1 code implementation • 29 Oct 2019 • David Alvarez-Melis, Hal Daumé III, Jennifer Wortman Vaughan, Hanna Wallach
Interpretability is an elusive but highly sought-after characteristic of modern machine learning methods.
1 code implementation • NeurIPS 2019 • Aaron Schein, Scott W. Linderman, Mingyuan Zhou, David M. Blei, Hanna Wallach
This paper presents the Poisson-randomized gamma dynamical system (PRGDS), a model for sequentially observed count tensors that encodes a strong inductive bias toward sparsity and burstiness.
no code implementations • 4 Jul 2019 • Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris
AI technologies have the potential to dramatically impact the lives of people with disabilities (PWD).
no code implementations • ACL 2019 • Alexander Hoyle, Wolf-Sonkin, Hanna Wallach, Isabelle Augenstein, Ryan Cotterell
Studying the ways in which language is gendered has long been an area of interest in sociolinguistics.
no code implementations • ACL 2019 • Ran Zmigrod, Sabrina J. Mielke, Hanna Wallach, Ryan Cotterell
Gender stereotypes are manifest in most of the world's languages and are consequently propagated or amplified by NLP systems.
no code implementations • NAACL 2019 • Alexey Romanov, Maria De-Arteaga, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Anna Rumshisky, Adam Tauman Kalai
In the context of mitigating bias in occupation classification, we propose a method for discouraging correlation between the predicted probability of an individual's true occupation and a word embedding of their name.
1 code implementation • NAACL 2019 • Alexander Hoyle, Lawrence Wolf-Sonkin, Hanna Wallach, Ryan Cotterell, Isabelle Augenstein
When assigning quantitative labels to a dataset, different methodologies may rely on different scales.
5 code implementations • 27 Jan 2019 • Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, Adam Tauman Kalai
We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives.
no code implementations • 13 Dec 2018 • Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daumé III, Miro Dudík, Hanna Wallach
The potential for machine learning (ML) systems to amplify social inequities and unfairness is receiving increasing popular and academic attention.
24 code implementations • 23 Mar 2018 • Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford
The machine learning community currently has no standardized process for documenting datasets, which can lead to severe consequences in high-stakes domains.
1 code implementation • 22 Mar 2018 • Aaron Schein, Zhiwei Steven Wu, Alexandra Schofield, Mingyuan Zhou, Hanna Wallach
We present a general method for privacy-preserving Bayesian inference in Poisson factorization, a broad class of models that includes some of the most widely used models in the social sciences.
3 code implementations • ICML 2018 • Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, Hanna Wallach
We present a systematic approach for achieving fairness in a binary classification setting.
1 code implementation • 21 Feb 2018 • Forough Poursabzi-Sangdeh, Daniel G. Goldstein, Jake M. Hofman, Jennifer Wortman Vaughan, Hanna Wallach
With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models.
1 code implementation • 19 Jan 2017 • Aaron Schein, Mingyuan Zhou, Hanna Wallach
We introduce a new dynamical system for sequentially observed multivariate count data.
1 code implementation • NeurIPS 2016 • Aaron Schein, Hanna Wallach, Mingyuan Zhou
This paper presents a dynamical system based on the Poisson-Gamma construction for sequentially observed multivariate count data.
no code implementations • NeurIPS 2016 • Giacomo Zanella, Brenda Betancourt, Hanna Wallach, Jeffrey Miller, Abbas Zaidi, Rebecca C. Steorts
Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points.
no code implementations • 7 Sep 2016 • Rahul Goel, Sandeep Soni, Naman Goyal, John Paparrizos, Hanna Wallach, Fernando Diaz, Jacob Eisenstein
Language change is a complex social phenomenon, revealing pathways of communication and sociocultural influence.
1 code implementation • 6 Jun 2016 • Aaron Schein, Mingyuan Zhou, David M. Blei, Hanna Wallach
We introduce Bayesian Poisson Tucker decomposition (BPTD) for modeling country--country interaction event data.
no code implementations • 2 Dec 2015 • Jeffrey Miller, Brenda Betancourt, Abbas Zaidi, Hanna Wallach, Rebecca C. Steorts
Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points.
1 code implementation • 10 Jun 2015 • Aaron Schein, John Paisley, David M. Blei, Hanna Wallach
We demonstrate that our model's predictive performance is better than that of standard non-negative tensor factorization methods.
no code implementations • 11 Nov 2014 • Fangjian Guo, Charles Blundell, Hanna Wallach, Katherine Heller
We present the Bayesian Echo Chamber, a new Bayesian generative model for social interaction data.
no code implementations • 15 Nov 2013 • Aaron Schein, Juston Moore, Hanna Wallach
Correlations between anomalous activity patterns can yield pertinent information about complex social processes: a significant deviation from normal behavior, exhibited simultaneously by multiple pairs of actors, provides evidence for some underlying relationship involving those pairs---i. e., a multilateral relation.