ActiveLab: Active Learning with Re-Labeling by Multiple Annotators

1 code implementation • 27 Jan 2023 • Hui Wen Goh, Jonas Mueller

It is thus common to employ multiple annotators to label data with some overlap between their examples.

Active Learning

CROWDLAB: Supervised learning to infer consensus labels and quality scores for data with multiple annotators

2 code implementations • 13 Oct 2022 • Hui Wen Goh, Ulyana Tkachenko, Jonas Mueller

For analyzing such data, we introduce CROWDLAB, a straightforward approach to utilize any trained classifier to estimate: (1) A consensus label for each example that aggregates the available annotations; (2) A confidence score for how likely each consensus label is correct; (3) A rating for each annotator quantifying the overall correctness of their labels.