Search Results for author:
Found 32 papers, 10 papers with code
Phi-4-reasoning Technical Report
We introduce Phi-4-reasoning, a 14-billion parameter reasoning model that achieves strong performance on complex reasoning tasks.
Inference-Time Scaling for Complex Tasks: Where We Stand and What Lies Ahead
Results from multiple independent runs with conventional models using perfect verifiers show that, for some tasks, these models can achieve performance close to the average performance of today's most advanced reasoning models.
MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation
Vision-language models (VLMs) are highly effective but often underperform on specialized tasks; for example, Llava-1. 5 struggles with chart and diagram understanding due to scarce task-specific training data.
Attention Speaks Volumes: Localizing and Mitigating Bias in Language Models
We believe that analyzing attention is also crucial for understanding bias, as it provides insight into how the LLM distributes its focus across different entities and how this contributes to biased decisions.
BENCHAGENTS: Automated Benchmark Creation with Agent Interaction
Evaluations are limited by benchmark availability.
Unearthing Skill-Level Insights for Understanding Trade-Offs of Foundation Models
With models getting stronger, evaluations have grown more complex, testing multiple skills in one benchmark and even in the same instance at once.
Improving Instruction-Following in Language Models through Activation Steering
Our experiments across four models demonstrate how we can use the activation vectors to guide models to follow constraints even without explicit instructions and to enhance performance when instructions are present.
Eureka: Evaluating and Understanding Large Foundation Models
Second, we introduce Eureka-Bench as an extensible collection of benchmarks testing capabilities that (i) are still challenging for state-of-the-art models and (ii) represent fundamental but overlooked language and multimodal capabilities.
Understanding Information Storage and Transfer in Multi-modal Large Language Models
Understanding the mechanisms of information storage and transfer in Transformer-based models is important for driving model understanding progress.
Introducing v0.5 of the AI Safety Benchmark from MLCommons
We created a new taxonomy of 13 hazard categories, of which 7 have tests in the v0. 5 benchmark.
Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models
We then compare the few-shot learning performance of LLMs on datasets that were seen during training to the performance on datasets released after training.
KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval
Motivated by rising concerns around factual incorrectness and hallucinations of LLMs, we present KITAB, a new dataset for measuring constraint satisfaction abilities of language models.
Diversity of Thought Improves Reasoning Abilities of LLMs
Large language models (LLMs) are documented to struggle in settings that require complex reasoning.
Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models
We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text.
Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning
Spurious correlations that degrade model generalization or lead the model to be right for the wrong reasons are one of the main robustness concerns for real-world deployments.
Social Biases through the Text-to-Image Generation Lens
In this paper, we take a multi-dimensional approach to studying and quantifying common social biases as reflected in the generated images, by focusing on how occupations, personality traits, and everyday situations are depicted across representations of (perceived) gender, age, race, and geographical location.
Benchmarking Spatial Relationships in Text-to-Image Generation
We investigate the ability of T2I models to generate correct spatial relationships among objects and present VISOR, an evaluation metric that captures how accurately the spatial relationship described in text is generated in the image.
Advancing Human-AI Complementarity: The Impact of User Expertise and Algorithmic Tuning on Joint Decision Making
In addition, we found that users' perception of the AI's performance relative on their own also had a significant impact on whether their accuracy improved when given AI recommendations.
Who Goes First? Influences of Human-AI Workflow on Decision Making in Clinical Imaging
A critical aspect of interaction design for AI-assisted human decision making are policies about the display and sequencing of AI inferences within larger decision-making workflows.
Investigations of Performance and Bias in Human-AI Teamwork in Hiring
In AI-assisted decision-making, effective hybrid (human-AI) teamwork is not solely dependent on AI performance alone, but also on its impact on human decision-making.
Hierarchical Analysis of Visual COVID-19 Features from Chest Radiographs
Chest radiography has been a recommended procedure for patient triaging and resource management in intensive care units (ICUs) throughout the COVID-19 pandemic.
Understanding Failures of Deep Networks via Robust Feature Extraction
Traditional evaluation metrics for learned models that report aggregate scores over a test set are insufficient for surfacing important and informative patterns of failure over features and instances.
An Empirical Analysis of Backward Compatibility in Machine Learning Systems
In many applications of machine learning (ML), updates are performed with the goal of enhancing model performance.
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance
However, prior studies observed improvements from explanations only when the AI, alone, outperformed both the human and the best team.
Is the Most Accurate AI the Best Teammate? Optimizing AI for Teamwork
To optimize the team performance for this setting we maximize the team's expected utility, expressed in terms of the quality of the final decision, cost of verifying, and individual accuracies of people and machines.
SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions
We quantify the extent to which this phenomenon occurs by creating a new Reasoning split of the VQA dataset and collecting VQA-introspect, a new dataset1 which consists of 238K new perception questions which serve as sub questions corresponding to the set of perceptual tasks needed to effectively answer the complex reasoning questions in the Reasoning split.
What You See Is What You Get? The Impact of Representation Criteria on Human Bias in Hiring
Although systematic biases in decision-making are widely documented, the ways in which they emerge from different sources is less understood.
A Case for Backward Compatibility for Human-AI Teams
We introduce the notion of the compatibility of an AI update with prior user experience and present methods for studying the role of compatibility in human-AI teams.
Metareasoning in Modular Software Systems: On-the-Fly Configuration using Reinforcement Learning with Rich Contextual Representations
We address the opportunity to maximize the utility of an overall computing system by employing reinforcement learning to guide the configuration of the set of interacting modules that comprise the system.
Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure
We present Pandora, a set of hybrid human-machine methods and tools for describing and explaining system failures.
On Human Intellect and Machine Failures: Troubleshooting Integrative Machine Learning Systems
We study the problem of troubleshooting machine learning systems that rely on analytical pipelines of distinct components.
Crowd Access Path Optimization: Diversity Matters
Based on this intuitive idea, we introduce the Access Path Model (APM), a novel crowd model that leverages the notion of access paths as an alternative way of retrieving information.
