Discrepancy Ratio: Evaluating Model Performance When Even Experts Disagree on the Truth

no code implementations • ICLR 2020 • Igor Lovchinsky, Alon Daks, Israel Malkin, Pouya Samangouei, Ardavan Saeedi, Yang Liu, Swami Sankaranarayanan, Tomer Gafner, Ben Sternlieb, Patrick Maher, Nathan Silberman

In most machine learning tasks unambiguous ground truth labels can easily be acquired.

BIG-bench Machine Learning Binary Classification