Search Results for author:
Found 7 papers, 0 papers with code
DataPerf: Benchmarks for Data-Centric AI Development
Machine learning (ML) research has generally focused on models, while the most prominent datasets have been employed for everyday ML tasks without regard for the breadth, difficulty, and faithfulness of these datasets to the underlying problem.
k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations
In these instances, the data reliability is under-reported, and a proposed k-rater reliability (kRR) should be used as the correct data reliability for aggregated datasets.
Data Excellence for AI: Why Should You Care
The efficacy of machine learning (ML) models depends on both algorithms and data.
Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability
When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012).
Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability
We present a new approach to interpreting IRR that is empirical and contextualized.
Metrology for AI: From Benchmarks to Instruments
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations.
