Search Results for author:
Found 16 papers, 3 papers with code
Medical Large Language Model Benchmarks Should Prioritize Construct Validity
In the psychological testing literature, "construct validity" refers to the ability of a test to measure an underlying "construct", that is the actual conceptual target of evaluation.
Toward an Evaluation Science for Generative AI Systems
There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts.
From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms
When an individual reports a negative interaction with some system, how can their personal experience be contextualized within broader patterns of system behavior?
The Data Addition Dilemma
In many machine learning for healthcare tasks, standard datasets are constructed by amassing data across many, often fundamentally dissimilar, sources.
Concrete Problems in AI Safety, Revisited
As AI systems proliferate in society, the AI community is increasingly preoccupied with the concept of AI Safety, namely the prevention of failures due to accidents that arise from an unanticipated departure of a system's behavior from designer intent in AI deployment.
REFORMS: Reporting Standards for Machine Learning Based Science
Machine learning (ML) methods are proliferating in scientific research.
Organizational Governance of Emerging Technologies: AI Adoption in Healthcare
This is one of the most detailed qualitative analyses to date of the current governance structures and processes involved in AI adoption by health systems in the United States.
AI and the Everything in the Whole Wide World Benchmark
There is a tendency across different subfields in AI to valorize a small collection of influential benchmarks.
About Face: A Survey of Facial Recognition Evaluation
We survey over 100 face datasets constructed between 1976 to 2019 of 145 million images of over 17 million subjects from a range of sources, demographics and conditions.
Data and its (dis)contents: A survey of dataset development and use in machine learning research
Datasets have played a foundational role in the advancement of machine learning research.
Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing
Although essential to revealing biased performance, well intentioned attempts at algorithmic auditing can have effects that may harm the very populations these measures are meant to protect.
Computers and Society
Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing
Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms.
Computers and Society
ABOUT ML: Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles
We present the "Annotation and Benchmarking on Understanding and Transparency of Machine Learning Lifecycles" (ABOUT ML) project as an initiative to operationalize ML transparency and work towards a standard ML documentation practice.
On the Legal Compatibility of Fairness Definitions
Past literature has been effective in demonstrating ideological gaps in machine learning (ML) fairness definitions when considering their use in complex socio-technical systems.
Model Cards for Model Reporting
Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information.
