Annotator Response Distributions as a Sampling Frame

no code implementations • NLPerspectives (LREC) 2022 • Christopher Homan, Tharindu Cyril Weerasooriya, Lora Aroyo, Chris Welty

Annotator disagreement is often dismissed as noise or the result of poor annotation process quality.

Embedding Semantic Taxonomies

no code implementations • COLING 2020 • Alyssa Lees, Chris Welty, Shubin Zhao, Jacek Korycki, Sara Mc Carthy

A common step in developing an understanding of a vertical domain, e. g. shopping, dining, movies, medicine, etc., is curating a taxonomy of categories specific to the domain.

Collaborative Filtering

Metrology for AI: From Benchmarks to Instruments

1 code implementation • 5 Nov 2019 • Chris Welty, Praveen Paritosh, Lora Aroyo

In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations.

Word Similarity

What is Fair? Exploring Pareto-Efficiency for Fairness Constrained Classifiers

no code implementations • 30 Oct 2019 • Ananth Balashankar, Alyssa Lees, Chris Welty, Lakshminarayanan Subramanian

The potential for learned models to amplify existing societal biases has been broadly recognized.

Fairness

A Crowdsourced Frame Disambiguation Corpus with Ambiguity

1 code implementation • NAACL 2019 • Anca Dumitrache, Lora Aroyo, Chris Welty

We present a resource for the task of FrameNet semantic frame disambiguation of over 5, 000 word-sentence pairs from the Wikipedia corpus.

LEMMA

Crowdsourcing Semantic Label Propagation in Relation Classification

1 code implementation • WS 2018 • Anca Dumitrache, Lora Aroyo, Chris Welty

Distant supervision is a popular method for performing relation extraction from text that is known to produce noisy labels.

Classification General Classification +1

CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

2 code implementations • 18 Aug 2018 • Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, Chris Welty

However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples.

Human-Computer Interaction Social and Information Networks

Capturing Ambiguity in Crowdsourcing Frame Disambiguation

1 code implementation • 1 May 2018 • Anca Dumitrache, Lora Aroyo, Chris Welty

FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words.

False Positive and Cross-relation Signals in Distant Supervision Data

1 code implementation • 14 Nov 2017 • Anca Dumitrache, Lora Aroyo, Chris Welty

Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair, then sentences that contain that pair are likely to express the relation.

General Classification Relation Classification

Crowdsourcing Ground Truth for Medical Relation Extraction

1 code implementation • 9 Jan 2017 • Anca Dumitrache, Lora Aroyo, Chris Welty

Cognitive computing systems require human labeled data for evaluation, and often for training.

Medical Relation Extraction Relation Extraction

Long-Distance Time-Event Relation Extraction

no code implementations • IJCNLP 2013 • Aless Moschitti, ro, Siddharth Patwardhan, Chris Welty

Event Relation Extraction Reading Comprehension +1