Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regression

no code implementations • 13 Mar 2025 • Hooman Shahrokhi, Devjeet Raj Roy, Yan Yan, Venera Arnaoudova, Janaradhan Rao Doppa

We consider the problem of generating valid and small prediction sets by sampling outputs (e. g., software code and natural language text) from a black-box deep generative model for a given input (e. g., textual prompt).

Code Generation Conformal Prediction +3

GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models

2 code implementations • 7 Oct 2024 • Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, Mehrdad Farajtabar

While the performance of LLMs on GSM8K has significantly improved in recent years, it remains unclear whether their mathematical reasoning capabilities have genuinely advanced, raising questions about the reliability of the reported metrics.

GSM8K Logical Reasoning +1

Duo-LLM: A Framework for Studying Adaptive Computation in Large Language Models

no code implementations • 1 Oct 2024 • Keivan Alizadeh, Iman Mirzadeh, Hooman Shahrokhi, Dmitry Belenko, Frank Sun, Minsik Cho, Mohammad Hossein Sekhavat, Moin Nabi, Mehrdad Farajtabar

Large Language Models (LLMs) typically generate outputs token by token using a fixed compute budget, leading to inefficient resource utilization.