Search Results for author:
Found 82 papers, 35 papers with code
Robust Black Box Explanations Under Distribution Shift
As machine learning black boxes are increasingly being deployed in real-world applications, there has been a growing interest in developing post hoc explanations that summarize the behaviors of these black box models.
How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior
In this paper, we conduct an empirical study on how memory management choices impact the LLM agents' behavior, especially their long-term performance.
Interpretability Illusions with Sparse Autoencoders: Evaluating Robustness of Concept Representations
Sparse autoencoders (SAEs) are commonly used to interpret the internal activations of large language models (LLMs) by mapping them to human-interpretable concept representations.
Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models
Large Reasoning Models (LRMs) have significantly enhanced their capabilities in complex problem-solving by introducing a thinking draft that enables multi-path Chain-of-Thought explorations before producing final answers.
Soft Best-of-n Sampling for Model Alignment
BoN yields high reward values in practice at a distortion cost, as measured by the KL-divergence between the sampled and original distribution.
How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidence
Post-training is essential for the success of large language models (LLMs), transforming pre-trained base models into more useful and aligned post-trained models.
Towards Interpretable Soft Prompts
Instead, our framework inspires a new direction of trainable prompting methods that explicitly optimizes for interpretability.
Detecting LLM-Generated Peer Reviews
We also empirically find that our approach is resilient to common reviewer defenses, and that the bounds on error rates in our statistical tests hold in practice.
Generalizing Trust: Weak-to-Strong Trustworthiness in Language Models
A key phenomenon known as weak-to-strong generalization - where a strong model trained on a weak model's outputs surpasses the weak model in task performance - has gained significant attention.
On the Impact of Fine-Tuning on Chain-of-Thought Reasoning
Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities that find applications across diverse domains.
Towards Unifying Interpretability and Control: Evaluation via Intervention
We introduce two new evaluation metrics: intervention success rate and the coherence-intervention tradeoff, designed to measure the accuracy of explanations and their utility in controlling model behavior.
Generalized Group Data Attribution
Data Attribution (DA) methods quantify the influence of individual training data points on model outputs and have broad applications such as explainability, data selection, and noisy label identification.
Quantifying Generalization Complexity for Large Language Models
Scylla disentangles generalization from memorization via assessing model performance on both in-distribution (ID) and out-of-distribution (OOD) data through 20 tasks across 5 levels of complexity.
Learning Recourse Costs from Pairwise Feature Comparisons
We demonstrate the efficient learning of individual feature costs using MAP estimates, and show that these non-exhaustive human surveys, which do not necessarily contain data for each feature pair comparison, are sufficient to learn an exhaustive set of feature costs, where each feature is associated with a modification cost.
All Roads Lead to Rome? Exploring Representational Similarities Between Latent Spaces of Generative Image Models
Do different generative image models secretly learn similar underlying representations?
Operationalizing the Blueprint for an AI Bill of Rights: Recommendations for Practitioners, Researchers, and Policy Makers
In this write-up, we address this shortcoming by providing an accessible overview of existing literature related to operationalizing regulatory principles.
On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models
In this work, we explore the promise of three broad approaches commonly employed to steer the behavior of LLMs to enhance the faithfulness of the CoT reasoning generated by LLMs: in-context learning, fine-tuning, and activation editing.
Interpretability Needs a New Paradigm
At present, interpretability is divided into two paradigms: the intrinsic paradigm, which believes that only models designed to be explained can be explained, and the post-hoc paradigm, which believes that black-box models can be explained.
More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness
The trustworthiness of Large Language Models (LLMs) refers to the extent to which their outputs are reliable, safe, and ethically aligned, and it has become a crucial consideration alongside their cognitive performance.
Manipulating Large Language Models to Increase Product Visibility
We demonstrate that adding a strategic text sequence (STS) -- a carefully crafted message -- to a product's information page can significantly increase its likelihood of being listed as the LLM's top recommendation.
Data Poisoning Attacks on Off-Policy Policy Evaluation Methods
Off-policy Evaluation (OPE) methods are a crucial tool for evaluating policies in high-stakes domains such as healthcare, where exploration is often infeasible, unethical, or expensive.
MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models
As large language models (LLMs) develop increasingly sophisticated capabilities and find applications in medical settings, it becomes important to assess their medical safety due to their far-reaching implications for personal and public health, patient safety, and human rights.
Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems
Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation.
OpenHEXAI: An Open-Source Framework for Human-Centered Evaluation of Explainable Machine Learning
However, properly evaluating the effectiveness of the XAI methods inevitably requires the involvement of human subjects, and conducting human-centered benchmarks is challenging in a number of ways: designing and implementing user studies is complex; numerous design choices in the design space of user study lead to problems of reproducibility; and running user studies can be challenging and even daunting for machine learning researchers.
Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)
In this work, we show that the semantic structure of CLIP's latent space can be leveraged to provide interpretability, allowing for the decomposition of representations into semantic concepts.
Towards Uncovering How Large Language Model Works: An Explainability Perspective
Large language models (LLMs) have led to breakthroughs in language tasks, yet the internal mechanisms that enable their remarkable generalization and reasoning abilities remain opaque.
Understanding the Effects of Iterative Prompting on Truthfulness
The development of Large Language Models (LLMs) has notably transformed numerous sectors, offering impressive text generation capabilities.
Faithfulness vs. Plausibility: On the (Un)Reliability of Explanations from Large Language Models
We highlight that the current trend towards increasing the plausibility of explanations, primarily driven by the demand for user-friendly interfaces, may come at the cost of diminishing their faithfulness.
A Study on the Calibration of In-context Learning
Accurate uncertainty quantification is crucial for the safe deployment of machine learning models, and prior research has demonstrated improvements in the calibration of modern language models (LMs).
Quantifying Uncertainty in Natural Language Explanations of Large Language Models
In this work, we make one of the first attempts at quantifying the uncertainty in explanations of LLMs.
Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classifications
Recent literature has suggested the potential of using large language models (LLMs) to make classifications for tabular tasks.
In-Context Unlearning: Language Models as Few Shot Unlearners
Machine unlearning, the study of efficiently removing the impact of specific training instances on a model, has garnered increased attention in recent years due to regulatory guidelines such as the \emph{Right to be Forgotten}.
In-Context Explainers: Harnessing LLMs for Explaining Black Box Models
Despite their effectiveness in enhancing the performance of LLMs on diverse language and tabular tasks, these methods have not been thoroughly explored for their potential to generate post hoc explanations.
Explainable artificial intelligence
Explainable Artificial Intelligence (XAI)
+2
On the Trade-offs between Adversarial Robustness and Actionable Explanations
As machine learning models are increasingly being employed in various high-stakes settings, it becomes important to ensure that predictions of these models are not only adversarially robust, but also readily explainable to relevant stakeholders.
Certifying LLM Safety against Adversarial Prompting
We defend against three attack modes: i) adversarial suffix, where an adversarial sequence is appended at the end of a harmful prompt; ii) adversarial insertion, where the adversarial sequence is inserted anywhere in the middle of the prompt; and iii) adversarial infusion, where adversarial tokens are inserted at arbitrary positions in the prompt, not necessarily as a contiguous block.
Accurate, Explainable, and Private Models: Providing Recourse While Minimizing Training Data Leakage
We present two novel methods to generate differentially private recourse: Differentially Private Model (DPM) and Laplace Recourse (LR).
Discriminative Feature Attributions: Bridging Post Hoc Explainability and Inherent Interpretability
This strategy naturally combines the ease of use of post hoc explanations with the faithfulness of inherently interpretable models.
Characterizing Data Point Vulnerability via Average-Case Robustness
Studying the robustness of machine learning models is important to ensure consistent model behaviour across real-world settings.
Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions
Chain-of-thought (CoT) prompting has been shown to empirically improve the accuracy of large language models (LLMs) on various question answering tasks.
On Minimizing the Impact of Dataset Shifts on Actionable Explanations
To this end, we conduct rigorous theoretical analysis to demonstrate that model curvature, weight decay parameters while training, and the magnitude of the dataset shift are key factors that determine the extent of explanation (in)stability.
Consistent Explanations in the Face of Model Indeterminacy via Ensembling
This work addresses the challenge of providing consistent explanations for predictive models in the presence of model indeterminacy, which arises due to the existence of multiple (nearly) equally well-performing models for a given dataset and task.
Word-Level Explanations for Analyzing Bias in Text-to-Image Models
We introduce a method for computing scores for each word in the prompt; these scores represent its influence on biases in the model's output.
Post Hoc Explanations of Language Models Can Improve Language Models
Large Language Models (LLMs) have demonstrated remarkable capabilities in performing complex tasks.
Towards Bridging the Gaps between the Right to Explanation and the Right to be Forgotten
The Right to Explanation and the Right to be Forgotten are two important principles outlined to regulate algorithmic decision making and data usage in real-world applications.
On the Privacy Risks of Algorithmic Recourse
As predictive models are increasingly being employed to make consequential decisions, there is a growing emphasis on developing techniques that can provide algorithmic recourse to affected individuals.
Towards Robust Off-Policy Evaluation via Human Inputs
When deployment environments are expected to undergo changes (that is, dataset shifts), it is important for OPE methods to perform robust evaluation of the policies amidst such changes.
Evaluating Explainability for Graph Neural Networks
As post hoc explanations are increasingly used to understand the behavior of graph neural networks (GNNs), it becomes crucial to evaluate the quality and reliability of GNN explanations.
TalkToModel: Explaining Machine Learning Models with Interactive Natural Language Conversations
In real-world evaluations with humans, 73% of healthcare workers (e. g., doctors and nurses) agreed they would use TalkToModel over baseline point-and-click systems for explainability in a disease prediction task, and 85% of ML professionals agreed TalkToModel was easier to use for computing explanations.
OpenXAI: Towards a Transparent Evaluation of Model Explanations
OpenXAI comprises of the following key components: (i) a flexible synthetic data generator and a collection of diverse real-world datasets, pre-trained models, and state-of-the-art feature attribution methods, and (ii) open-source implementations of eleven quantitative metrics for evaluating faithfulness, stability (robustness), and fairness of explanation methods, in turn providing comparisons of several explanation methods across a wide variety of metrics, models, and datasets.
Efficiently Training Low-Curvature Neural Networks
To achieve this, we minimize a data-independent upper bound on the curvature of a neural network, which decomposes overall curvature in terms of curvatures and slopes of its constituent layers.
A Human-Centric Take on Model Monitoring
Predictive models are increasingly used to make various consequential decisions in high-stakes domains such as healthcare, finance, and policy.
Which Explanation Should I Choose? A Function Approximation Perspective to Characterizing Post Hoc Explanations
By bringing diverse explanation methods into a common framework, this work (1) advances the conceptual understanding of these methods, revealing their shared local function approximation objective, properties, and relation to one another, and (2) guides the use of these methods in practice, providing a principled approach to choose among methods and paving the way for the creation of new ones.
Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
We then leverage these properties to propose a novel evaluation framework which can quantitatively measure disparities in the quality of explanations output by state-of-the-art methods.
Rethinking Stability for Attribution-based Explanations
As attribution-based explanation methods are increasingly used to establish model trustworthiness in high-stakes situations, it is critical to ensure that these explanations are stable, e. g., robust to infinitesimal perturbations to an input.
Probabilistically Robust Recourse: Navigating the Trade-offs between Costs and Robustness in Algorithmic Recourse
To this end, we propose a novel objective function which simultaneously minimizes the gap between the achieved (resulting) and desired recourse invalidation rates, minimizes recourse costs, and also ensures that the resulting recourse achieves a positive model prediction.
Rethinking Explainability as a Dialogue: A Practitioner's Perspective
Overall, we hope our work serves as a starting place for researchers and engineers to design interactive explainability systems.
The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
In addition, we carry out an online user study with data scientists to understand how they resolve the aforementioned disagreements.
What will it take to generate fairness-preserving explanations?
In situations where explanations of black-box models may be useful, the fairness of the black-box is also often a relevant concern.
Feature Attributions and Counterfactual Explanations Can Be Manipulated
As machine learning models are increasingly used in critical decision-making settings (e. g., healthcare, finance), there has been a growing emphasis on developing methods to explain model predictions.
Exploring Counterfactual Explanations Through the Lens of Adversarial Examples: A Theoretical and Empirical Analysis
As machine learning (ML) models become more widely deployed in high-stakes applications, counterfactual explanations have emerged as key tools for providing actionable model explanations in practice.
Probing GNN Explainers: A Rigorous Theoretical and Empirical Analysis of GNN Explanation Methods
As Graph Neural Networks (GNNs) are increasingly being employed in critical real-world applications, several methods have been proposed in recent literature to explain the predictions of these models.
Counterfactual Explanations Can Be Manipulated
In this work, we introduce the first framework that describes the vulnerabilities of counterfactual explanations and shows how they can be manipulated.
Learning Under Adversarial and Interventional Shifts
Most of the existing work focuses on optimizing for either adversarial shifts or interventional shifts.
Towards Robust and Reliable Algorithmic Recourse
To address this problem, we propose a novel framework, RObust Algorithmic Recourse (ROAR), that leverages adversarial training for finding recourses that are robust to model shifts.
Towards a Unified Framework for Fair and Stable Graph Representation Learning
In this work, we establish a key connection between counterfactual fairness and stability and leverage it to propose a novel framework, NIFTY (uNIfying Fairness and stabiliTY), which can be used with any GNN to learn fair and stable representations.
Towards the Unification and Robustness of Perturbation and Gradient Based Explanations
As machine learning black boxes are increasingly being deployed in critical domains such as healthcare and criminal justice, there has been a growing emphasis on developing techniques for explaining these black boxes in a post hoc manner.
Algorithmic Recourse in the Wild: Understanding the Impact of Data and Model Shifts
Our theoretical results establish a lower bound on the probability of recourse invalidation due to model shifts, and show the existence of a tradeoff between this invalidation probability and typical notions of "cost" minimized by modern recourse generation algorithms.
Does Fair Ranking Improve Minority Outcomes? Understanding the Interplay of Human and Algorithmic Biases in Online Hiring
In this work, we analyze various sources of gender biases in online hiring platforms, including the job context and inherent biases of employers and establish how these factors interact with ranking algorithms to affect hiring decisions.
Robust and Stable Black Box Explanations
To the best of our knowledge, this work makes the first attempt at generating post hoc explanations that are robust to a general class of adversarial perturbations that are of practical interest.
Learning Models for Actionable Recourse
As machine learning models are increasingly deployed in high-stakes domains such as legal and financial decision-making, there has been growing interest in post-hoc methods for generating counterfactual explanations.
When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making
As machine learning (ML) models are increasingly being employed to assist human decision makers, it becomes critical to provide these decision makers with relevant inputs which can help them decide if and how to incorporate model predictions into their decision making.
Incorporating Interpretable Output Constraints in Bayesian Neural Networks
Domains where supervised models are deployed often come with task-specific constraints, such as prior expert knowledge on the ground-truth function, or desiderata like safety and fairness.
Beyond Individualized Recourse: Interpretable and Interactive Summaries of Actionable Recourses
As predictive models are increasingly being deployed in high-stakes decision-making, there has been a lot of interest in developing algorithms which can provide recourses to affected individuals.
Reliable Post hoc Explanations: Modeling Uncertainty in Explainability
In this paper, we address the aforementioned challenges by developing a novel Bayesian framework for generating local explanations along with their associated uncertainty.
Fair Influence Maximization: A Welfare Optimization Approach
Under this framework, the trade-off between fairness and efficiency can be controlled by a single inequality aversion design parameter.
"How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
Our work is the first to empirically establish how user trust in black box models can be manipulated via misleading explanations.
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Our approach can be used to scaffold any biased classifier in such a way that its predictions on the input data distribution still remain biased, but the post hoc explanations of the scaffolded classifier look innocuous.
Interpretable & Explorable Approximations of Black Box Models
To the best of our knowledge, this is the first approach which can produce global explanations of the behavior of any given black box model through joint optimization of unambiguity, fidelity, and interpretability, while also allowing users to explore model behavior based on their preferences.
Confusions over Time: An Interpretable Bayesian Model to Characterize Trends in Decision Making
We propose Confusions over Time (CoT), a novel generative framework which facilitates a multi-granular analysis of the decision making process.
Learning Cost-Effective and Interpretable Regimes for Treatment Recommendation
We formulate this as a problem of learning a decision list -- a sequence of if-then-else rules -- which maps characteristics of subjects (eg., diagnostic test results of patients) to treatments.
Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Exploration
Predictive models deployed in the real world may assign incorrect labels to instances with high confidence.
Learning Cost-Effective Treatment Regimes using Markov Decision Processes
We formulate this as a problem of learning a decision list -- a sequence of if-then-else rules -- which maps characteristics of subjects (eg., diagnostic test results of patients) to treatments.
