Search Results for author: Praveen Paritosh

Found 7 papers, 0 papers with code

k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations

no code implementations ACL 2022 Ka Wong, Praveen Paritosh

In these instances, the data reliability is under-reported, and a proposed k-rater reliability (kRR) should be used as the correct data reliability for aggregated datasets.

Natural Language Processing

Data Excellence for AI: Why Should You Care

no code implementations19 Nov 2021 Lora Aroyo, Matthew Lease, Praveen Paritosh, Mike Schaekermann

The efficacy of machine learning (ML) models depends on both algorithms and data.

Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability

no code implementations ACL 2021 Ka Wong, Praveen Paritosh, Lora Aroyo

When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012).

Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability

no code implementations11 Jun 2021 Ka Wong, Praveen Paritosh, Lora Aroyo

We present a new approach to interpreting IRR that is empirical and contextualized.

Metrology for AI: From Benchmarks to Instruments

no code implementations5 Nov 2019 Chris Welty, Praveen Paritosh, Lora Aroyo

In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations.

Word Similarity

Cannot find the paper you are looking for? You can Submit a new open access paper.