Search Results for author:
Found 6 papers, 6 papers with code
HCAST: Human-Calibrated Autonomy Software Tasks
To understand and predict the societal impacts of highly autonomous AI systems, we need benchmarks with grounding, i. e., metrics that directly connect AI performance to real-world effects we care about.
Measuring AI Ability to Complete Long Tasks
Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear.
Training Language Models to Win Debates with Self-Play Improves Judge Accuracy
We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play.
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry.
Debate Helps Supervise Unreliable Experts
Comparing debate to a baseline we call consultancy, where a single expert argues for only one answer which is correct half of the time, we find that debate performs significantly better, with 84% judge accuracy compared to consultancy's 74%.
Classification with Strategically Withheld Data
To this end, we present {\sc IC-LR}, a modification of Logistic Regression that removes the incentive to strategically drop features.
