PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models

no code implementations • 22 Nov 2024 • Gerald Friedland, Xin Huang, Yueying Cui, Vishaal Kapoor, Ashish Khetan, Sanjiv Das

The method and metric enables users to rank generative language models for quality of responses, so as to make a selection of the best model for a given task.

Diverse Counterfactual Explanations for Anomaly Detection in Time Series

no code implementations • 21 Mar 2022 • Deborah Sulem, Michele Donini, Muhammad Bilal Zafar, Francois-Xavier Aubet, Jan Gasthaus, Tim Januschowski, Sanjiv Das, Krishnaram Kenthapadi, Cedric Archambeau

In this work we propose a model-agnostic algorithm that generates counterfactual ensemble explanations for time series anomaly detection models.

Anomaly Detection counterfactual +2

Amazon SageMaker Clarify: Machine Learning Bias Detection and Explainability in the Cloud

1 code implementation • 7 Sep 2021 • Michaela Hardt, Xiaoguang Chen, Xiaoyi Cheng, Michele Donini, Jason Gelman, Satish Gollaprolu, John He, Pedro Larroy, Xinyu Liu, Nick McCarthy, Ashish Rathi, Scott Rees, Ankit Siva, ErhYuan Tsai, Keerthan Vasist, Pinar Yilmaz, Muhammad Bilal Zafar, Sanjiv Das, Kevin Haas, Tyler Hill, Krishnaram Kenthapadi

We present Amazon SageMaker Clarify, an explainability feature for Amazon SageMaker that launched in December 2020, providing insights into data and ML models by identifying biases and explaining predictions.

Bias Detection BIG-bench Machine Learning +2

On the Lack of Robust Interpretability of Neural Text Classifiers

no code implementations • Findings (ACL) 2021 • Muhammad Bilal Zafar, Michele Donini, Dylan Slack, Cédric Archambeau, Sanjiv Das, Krishnaram Kenthapadi

With the ever-increasing complexity of neural language models, practitioners have turned to methods for understanding the predictions of these models.