Search Results for author:
Found 22 papers, 16 papers with code
Cost-of-Pass: An Economic Framework for Evaluating Language Models
We find that innovations in lightweight, large, and reasoning models have been essential for pushing the frontier in basic quantitative, knowledge-intensive, and complex quantitative tasks, respectively.
Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025
The results show that 27% of reviewers who received feedback updated their reviews, and over 12, 000 feedback suggestions from the agent were incorporated by those reviewers.
Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory
Overall, our findings present DC as a promising approach for augmenting LMs with persistent memory, bridging the divide between isolated inference events and the cumulative, experience-driven learning characteristic of human cognition.
Dynamical phases of short-term memory mechanisms in RNNs
Short-term memory is essential for cognitive processing, yet our understanding of its neural mechanisms remains unclear.
SiriuS: Self-improving Multi-agent Systems via Bootstrapped Reasoning
A key challenge in optimizing multi-agent systems is acquiring suitable training data for specialized agents.
A ghost mechanism: An analytical model of abrupt learning
\emph{Abrupt learning} is commonly observed in neural networks, where long plateaus in network performance are followed by rapid convergence to a desirable solution.
TextGrad: Automatic "Differentiation" via Text
Without modifying the framework, TextGrad improves the zero-shot accuracy of GPT-4o in Google-Proof Question Answering from $51\%$ to $55\%$, yields $20\%$ relative performance gain in optimizing LeetCode-Hard coding problem solutions, improves prompts for reasoning, designs new druglike small molecules with desirable in silico binding, and designs radiation oncology treatment plans with high specificity.
Ranked #5 on
on GPQA
How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis
We develop NegotiationArena: a flexible framework for evaluating and probing the negotiation abilities of LLM agents.
ChatGPT Exhibits Gender and Racial Biases in Acute Coronary Syndrome Management
One early application has been to medicine, where LLMs have been investigated to streamline clinical workflows and facilitate clinical analysis and decision-making.
KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval
Motivated by rising concerns around factual incorrectness and hallucinations of LLMs, we present KITAB, a new dataset for measuring constraint satisfaction abilities of language models.
Diversity of Thought Improves Reasoning Abilities of LLMs
Large language models (LLMs) are documented to struggle in settings that require complex reasoning.
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text.
Discover and Cure: Concept-aware Mitigation of Spurious Correlation
Deep neural networks often rely on spurious correlations to make predictions, which hinders generalization beyond training environments.
GPT detectors are biased against non-native English writers
In this study, we evaluate the performance of several widely-used GPT detectors using writing samples from native and non-native English writers.
Leveraging medical Twitter to build a visual–language foundation model for pathology AI
This is the largest public dataset for pathology images annotated with natural text.
SkinCon: A skin disease dataset densely annotated by domain experts for fine-grained model debugging and analysis
To provide a medical dataset densely annotated by domain experts with annotations useful across multiple disease processes, we developed SkinCon: a skin disease dataset densely annotated by dermatologists.
Holistic Evaluation of Language Models
We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
When and why vision-language models behave like bags-of-words, and what to do about it?
ARO consists of Visual Genome Attribution, to test the understanding of objects' properties; Visual Genome Relation, to test for relational understanding; and COCO & Flickr30k-Order, to test for order sensitivity.
Post-hoc Concept Bottleneck Models
When concept annotations are not available on the training data, we show that PCBM can transfer concepts from other datasets or from natural language descriptions of concepts via multimodal models.
Meaningfully Debugging Model Mistakes using Conceptual Counterfactual Explanations
Understanding and explaining the mistakes made by trained models is critical to many machine learning objectives, such as improving robustness, addressing concept drift, and mitigating biases.
ImageNet performance correlates with pose estimation robustness and generalization on out-of-domain data
Neural networks are highly effective tools for pose estimation.
Learning Maximally Predictive Prototypes in Multiple Instance Learning
In this work, we propose a simple model that provides permutation invariant maximally predictive prototype generator from a given dataset, which leads to interpretability of the solution and concrete insights to the nature and the solution of a problem.
