Search Results for author:
Found 20 papers, 4 papers with code
The Leaderboard Illusion
Providers like Google and OpenAI have received an estimated 19. 2% and 20. 4% of all data on the arena, respectively.
In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI
The widespread deployment of general-purpose AI (GPAI) systems introduces significant new risks.
International AI Safety Report
The first International AI Safety Report comprehensively synthesizes the current evidence on the capabilities, risks, and safety of advanced AI systems.
The Reality of AI and Biorisk
To accurately and confidently answer the question 'could an AI model or system increase biorisk', it is necessary to have both a sound theoretical threat model for how AI models or systems could increase biorisk and a robust method for testing that threat model.
Inference Scaling fLaws: The Limits of LLM Resampling with Imperfect Verifiers
The central thesis of this paper is that there is no free lunch for inference scaling: indefinite accuracy improvement through resampling can only be realized if the "verifier" (in this case, a set of unit tests) is perfect.
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility.
The 2024 Foundation Model Transparency Index
To characterize the status quo, the Foundation Model Transparency Index (FMTI) was launched in October 2023 to measure the transparency of leading foundation model developers.
AI Agents That Matter
We hope that the steps we introduce for addressing these shortcomings will spur the development of agents that are useful in the real world and not just accurate on benchmarks.
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications.
Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligence
Over the past year, there has been a robust debate about the benefits and risks of open sourcing foundation models.
A Safe Harbor for AI Evaluation and Red Teaming
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems.
On the Societal Impact of Open Foundation Models
To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk.
Foundation Model Transparency Reports
Foundation models are critical digital technologies with sweeping societal impact that necessitates transparency.
Promises and pitfalls of artificial intelligence for legal applications
Is AI set to redefine the legal profession?
The Foundation Model Transparency Index
We score 10 major foundation model developers (e. g. OpenAI, Google, Meta) against the 100 indicators to assess their transparency.
REFORMS: Reporting Standards for Machine Learning Based Science
Machine learning (ML) methods are proliferating in scientific research.
Leakage and the Reproducibility Crisis in ML-based Science
To investigate the impact of reproducibility errors and the efficacy of model info sheets, we undertake a reproducibility study in a field where complex ML models are believed to vastly outperform older statistical models such as Logistic Regression (LR): civil war prediction.
The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning
We conclude by discussing risks that arise when sources of errors are misdiagnosed and the need to acknowledge the role of human inductive biases in learning and reform.
Balanced News Using Constrained Bandit-based Personalization
We present a prototype for a news search engine that presents balanced viewpoints across liberal and conservative articles with the goal of de-polarizing content and allowing users to escape their filter bubble.
An Algorithmic Framework to Control Bias in Bandit-based Personalization
Personalization is pervasive in the online space as it leads to higher efficiency and revenue by allowing the most relevant content to be served to each user.
