Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine

1 code implementation • 28 Nov 2023 • Harsha Nori, Yin Tat Lee, Sheng Zhang, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, Eric Horvitz

We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks.

Electrical Engineering Experimental Design +3

A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications

no code implementations • 26 Oct 2023 • Ahmed Magooda, Alec Helyar, Kyle Jackson, David Sullivan, Chad Atalla, Emily Sheng, Dan Vann, Richard Edgar, Hamid Palangi, Roman Lutz, Hongliang Kong, Vincent Yun, Eslam Kamal, Federico Zarfati, Hanna Wallach, Sarah Bird, Mei Chen

We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services.

Fairlearn: Assessing and Improving Fairness of AI Systems

no code implementations • 29 Mar 2023 • Hilde Weerts, Miroslav Dudík, Richard Edgar, Adrin Jalali, Roman Lutz, Michael Madaio

Fairlearn is an open source project to help practitioners assess and improve fairness of artificial intelligence (AI) systems.

Fairness