Measuring AI Ability to Complete Long Tasks

1 code implementation • 18 Mar 2025 • Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan

Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear.

Logical Reasoning

Evaluating Language-Model Agents on Realistic Autonomous Tasks

no code implementations • 18 Dec 2023 • Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R. Lin, Hjalmar Wijk, Joel Burget, Aaron Ho, Elizabeth Barnes, Paul Christiano

We find that these language model agents can only complete the easiest tasks from this list, although they make some progress on the more challenging tasks.

Language Modeling Language Modelling +1

A Simulator for Hedonic Games

no code implementations • 26 Jun 2017 • Luke Harold Miles

Hedonic games are meant to model how coalitions of people form and break apart in the real world.

The Complexity of Campaigning

no code implementations • 20 Jun 2017 • Cory Siler, Luke Harold Miles, Judy Goldsmith

In "The Logic of Campaigning", Dean and Parikh consider a candidate making campaign statements to appeal to the voters.