Empirical Optimal Risk to Quantify Model Trustworthiness for Failure Detection

no code implementations • 6 Aug 2023 • Shuang Ao, Stefan Rueger, Advaith Siddharthan

We propose the Excess Area Under the Optimal RC Curve (E-AUoptRC), with the area in coverage from the optimal point to the full coverage.

Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration

1 code implementation • 6 Aug 2023 • Shuang Ao, Stefan Rueger, Advaith Siddharthan

Then we utilize the class-wise miscalibration score as a proxy to design a calibration technique that can tackle both over and under-confidence.

Confidence-Aware Calibration and Scoring Functions for Curriculum Learning

1 code implementation • 29 Jan 2023 • Shuang Ao, Stefan Rueger, Advaith Siddharthan

In this paper we integrate notions of model confidence and human confidence with label smoothing, respectively \textit{Model Confidence LS} and \textit{Human Confidence LS}, to achieve better model calibration and generalization.

text-classification Text Classification

Adverse Drug Reaction Classification With Deep Neural Networks

no code implementations • COLING 2016 • Trung Huynh, Yulan He, Alistair Willis, Stefan Rueger

On the Twitter dataset, all the NN architectures perform similarly.

Binary Classification Classification +1