Search Results for author:
Found 45 papers, 17 papers with code
Taxonomizing Representational Harms using Speech Act Theory
We present a framework, grounded in speech act theory (Austin, 1962), that conceptualizes representational harms caused by generative language systems as the perlocutionary effects (i. e., real-world impacts) of particular types of illocutionary acts (i. e., system behaviors).
Validating LLM-as-a-Judge Systems in the Absence of Gold Labels
High agreement rates between these gold labels and judge system ratings are then taken as a sign of good judge system performance.
Toward an Evaluation Science for Generative AI Systems
There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts.
Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice
We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy.
A Framework for Evaluating LLMs Under Task Indeterminacy
Large language model (LLM) evaluations often assume there is a single correct response -- a gold label -- for each item in the evaluation corpus.
A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services.
"One-Size-Fits-All"? Examining Expectations around What Constitute "Fair" or "Good" NLG System Behaviors
Fairness-related assumptions about what constitute appropriate NLG system behaviors range from invariance, where systems are expected to behave identically for social groups, to adaptation, where behaviors should instead vary across them.
FairPrism: Evaluating Fairness-Related Harms in Text Generation
It is critical to measure and mitigate fairness- related harms caused by AI text generation systems, including stereotyping and demeaning harms.
Measuring Representational Harms in Image Captioning
We propose multiple measurement techniques for each type of harm.
Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desiderata
Despite the fact that data documentation frameworks are often motivated from the perspective of responsible AI, participants did not make the connection between the questions that they were asked to answer and their responsible AI implications.
REAL ML: Recognizing, Exploring, and Articulating Limitations of Machine Learning Research
Transparency around limitations can improve the scientific rigor of research, help ensure appropriate interpretation of research findings, and make research claims more credible.
Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Support
Various tools and practices have been developed to support practitioners in identifying, assessing, and mitigating fairness-related harms caused by AI systems.
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets
Auditing NLP systems for computational harms like surfacing stereotypes is an elusive goal.
Doubly Non-Central Beta Matrix Factorization for DNA Methylation Data
We present a new non-negative matrix factorization model for $(0, 1)$ bounded-support data based on the doubly non-central beta (DNCB) distribution, a generalization of the beta distribution.
From Human Explanation to Model Interpretability: A Framework Based on Weight of Evidence
We take inspiration from the study of human explanation to inform the design and evaluation of interpretability methods in machine learning.
BIG-bench Machine Learning
Interpretable Machine Learning
+1
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs
Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple.
Language (Technology) is Power: A Critical Survey of ``Bias'' in NLP
We survey 146 papers analyzing {``}bias{''} in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing {``}bias{''} is an inherently normative process.
Language (Technology) is Power: A Critical Survey of "Bias" in NLP
We survey 146 papers analyzing "bias" in NLP systems, finding that their motivations are often vague, inconsistent, and lacking in normative reasoning, despite the fact that analyzing "bias" is an inherently normative process.
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs
We also find that there are statistically significant relationships between the grammatical genders of inanimate nouns and the verbs that take those nouns as direct objects, as indirect objects, and as subjects.
Measurement and Fairness
We argue that this contestedness underlies recent debates about fairness definitions: although these debates appear to be about different operationalizations, they are, in fact, debates about different theoretical understandings of fairness.
Quantifying the Semantic Core of Gender Systems
To that end, we use canonical correlation analysis to correlate the grammatical gender of inanimate nouns with an externally grounded definition of their lexical semantics.
Weight of Evidence as a Basis for Human-Oriented Explanations
Interpretability is an elusive but highly sought-after characteristic of modern machine learning methods.
Poisson-Randomized Gamma Dynamical Systems
This paper presents the Poisson-randomized gamma dynamical system (PRGDS), a model for sequentially observed count tensors that encodes a strong inductive bias toward sparsity and burstiness.
Toward Fairness in AI for People with Disabilities: A Research Roadmap
AI technologies have the potential to dramatically impact the lives of people with disabilities (PWD).
Unsupervised Discovery of Gendered Language through Latent-Variable Modeling
Studying the ways in which language is gendered has long been an area of interest in sociolinguistics.
Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
Gender stereotypes are manifest in most of the world's languages and are consequently propagated or amplified by NLP systems.
What's in a Name? Reducing Bias in Bios without Access to Protected Attributes
In the context of mitigating bias in occupation classification, we propose a method for discouraging correlation between the predicted probability of an individual's true occupation and a word embedding of their name.
Combining Sentiment Lexica with a Multi-View Variational Autoencoder
When assigning quantitative labels to a dataset, different methodologies may rely on different scales.
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
We present a large-scale study of gender bias in occupation classification, a task where the use of machine learning may lead to negative outcomes on peoples' lives.
Improving fairness in machine learning systems: What do industry practitioners need?
The potential for machine learning (ML) systems to amplify social inequities and unfairness is receiving increasing popular and academic attention.
Datasheets for Datasets
The machine learning community currently has no standardized process for documenting datasets, which can lead to severe consequences in high-stakes domains.
Locally Private Bayesian Inference for Count Models
We present a general method for privacy-preserving Bayesian inference in Poisson factorization, a broad class of models that includes some of the most widely used models in the social sciences.
A Reductions Approach to Fair Classification
We present a systematic approach for achieving fairness in a binary classification setting.
Manipulating and Measuring Model Interpretability
With machine learning models being increasingly used to aid decision making even in high-stakes domains, there has been a growing interest in developing interpretable models.
Poisson--Gamma Dynamical Systems
We introduce a new dynamical system for sequentially observed multivariate count data.
Poisson-Gamma dynamical systems
This paper presents a dynamical system based on the Poisson-Gamma construction for sequentially observed multivariate count data.
Flexible Models for Microclustering with Application to Entity Resolution
Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points.
The Social Dynamics of Language Change in Online Networks
Language change is a complex social phenomenon, revealing pathways of communication and sociocultural influence.
Bayesian Poisson Tucker Decomposition for Learning the Structure of International Relations
We introduce Bayesian Poisson Tucker decomposition (BPTD) for modeling country--country interaction event data.
Microclustering: When the Cluster Sizes Grow Sublinearly with the Size of the Data Set
Most generative models for clustering implicitly assume that the number of data points in each cluster grows linearly with the total number of data points.
Bayesian Poisson Tensor Factorization for Inferring Multilateral Relations from Sparse Dyadic Event Counts
We demonstrate that our model's predictive performance is better than that of standard non-negative tensor factorization methods.
The Bayesian Echo Chamber: Modeling Social Influence via Linguistic Accommodation
We present the Bayesian Echo Chamber, a new Bayesian generative model for social interaction data.
Inferring Multilateral Relations from Dynamic Pairwise Interactions
Correlations between anomalous activity patterns can yield pertinent information about complex social processes: a significant deviation from normal behavior, exhibited simultaneously by multiple pairs of actors, provides evidence for some underlying relationship involving those pairs---i. e., a multilateral relation.
