A2Perf: Real-World Autonomous Agents Benchmark

no code implementations • 4 Mar 2025 • Ikechukwu Uchendu, Jason Jabbour, Korneel Van den Berghe, Joel Runevic, Matthew Stewart, Jeffrey Ma, Srivatsan Krishnan, Izzeddin Gur, Austin Huang, Colton Bishop, Paige Bailey, Wenjie Jiang, Ebrahim M. Songhori, Sergio Guadarrama, Jie Tan, Jordan K. Terry, Aleksandra Faust, Vijay Janapa Reddi

However, there is a lack of benchmarking suites that define the environments, datasets, and metrics which can be used to provide a meaningful way for the community to compare progress on applying these methods to real-world problems.

Benchmarking Combinatorial Optimization +1

QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture

no code implementations • 3 Jan 2025 • Shvetank Prakash, Andrew Cheng, Jason Yik, Arya Tschand, Radhika Ghosal, Ikechukwu Uchendu, Jessica Quaye, Jeffrey Ma, Shreyas Grampurohit, Sofia Giannuzzi, Arnav Balyan, Fin Amin, Aadya Pipersenia, Yash Choudhary, Ankita Nayak, Amir Yazdanbakhsh, Vijay Janapa Reddi

Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research.

Benchmarking Question Answering

ArchGym: An Open-Source Gymnasium for Machine Learning Assisted Architecture Design

no code implementations • 15 Jun 2023 • Srivatsan Krishnan, Amir Yazdanbaksh, Shvetank Prakash, Jason Jabbour, Ikechukwu Uchendu, Susobhan Ghosh, Behzad Boroujerdian, Daniel Richins, Devashree Tripathy, Aleksandra Faust, Vijay Janapa Reddi

The ease of data collection and aggregation in ArchGym facilitates research in ML-aided architecture design space exploration.

Jump-Start Reinforcement Learning

1 code implementation • 5 Apr 2022 • Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, Karol Hausman

In addition, we provide an upper bound on the sample complexity of JSRL and show that with the help of a guide-policy, one can improve the sample complexity for non-optimism exploration methods from exponential in horizon to polynomial.

reinforcement-learning Reinforcement Learning +1

Improving Mild Cognitive Impairment Prediction via Reinforcement Learning and Dialogue Simulation

no code implementations • 18 Feb 2018 • Fengyi Tang, Kaixiang Lin, Ikechukwu Uchendu, Hiroko H. Dodge, Jiayu Zhou

Even though there is mild cognitive decline in MCI patients, they have normal overall cognition and thus is challenging to distinguish from normal aging.

reinforcement-learning Reinforcement Learning +1