Search Results for author: Lora Aroyo

Found 18 papers, 9 papers with code

Data Excellence for AI: Why Should You Care

no code implementations19 Nov 2021 Lora Aroyo, Matthew Lease, Praveen Paritosh, Mike Schaekermann

The efficacy of machine learning (ML) models depends on both algorithms and data.

Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability

no code implementations ACL 2021 Ka Wong, Praveen Paritosh, Lora Aroyo

When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012).

Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability

no code implementations11 Jun 2021 Ka Wong, Praveen Paritosh, Lora Aroyo

We present a new approach to interpreting IRR that is empirical and contextualized.

Metrology for AI: From Benchmarks to Instruments

no code implementations5 Nov 2019 Chris Welty, Praveen Paritosh, Lora Aroyo

In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations.

Word Similarity

A Crowdsourced Frame Disambiguation Corpus with Ambiguity

1 code implementation NAACL 2019 Anca Dumitrache, Lora Aroyo, Chris Welty

We present a resource for the task of FrameNet semantic frame disambiguation of over 5, 000 word-sentence pairs from the Wikipedia corpus.

Natural Language Processing

Homonym Detection For Humor Recognition In Short Text

no code implementations WS 2018 Sven van den Beukel, Lora Aroyo

In this paper, automatic homophone- and homograph detection are suggested as new useful features for humor recognition systems.

General Classification

Crowdsourcing Semantic Label Propagation in Relation Classification

1 code implementation WS 2018 Anca Dumitrache, Lora Aroyo, Chris Welty

Distant supervision is a popular method for performing relation extraction from text that is known to produce noisy labels.

Classification General Classification +1

CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

2 code implementations18 Aug 2018 Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, Chris Welty

However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples.

Human-Computer Interaction Social and Information Networks

Capturing Ambiguity in Crowdsourcing Frame Disambiguation

1 code implementation1 May 2018 Anca Dumitrache, Lora Aroyo, Chris Welty

FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words.

False Positive and Cross-relation Signals in Distant Supervision Data

1 code implementation14 Nov 2017 Anca Dumitrache, Lora Aroyo, Chris Welty

Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair, then sentences that contain that pair are likely to express the relation.

General Classification Relation Classification

Crowdsourcing Ground Truth for Medical Relation Extraction

1 code implementation9 Jan 2017 Anca Dumitrache, Lora Aroyo, Chris Welty

Cognitive computing systems require human labeled data for evaluation, and often for training.

Medical Relation Extraction Relation Extraction

Crowdsourcing Salient Information from News and Tweets

no code implementations LREC 2016 Oana Inel, Tommaso Caselli, Lora Aroyo

On the other hand, machines need to understand the information that is published in online data streams and generate concise and meaningful overviews.

GRaSP: A Multilayered Annotation Scheme for Perspectives

no code implementations LREC 2016 Chantal van Son, Tommaso Caselli, Antske Fokkens, Isa Maks, Roser Morante, Lora Aroyo, Piek Vossen

In the last decade, different aspects of linguistic encoding of perspectives have been targeted as separated phenomena through different annotation initiatives.

The VU Sound Corpus: Adding More Fine-grained Annotations to the Freesound Database

no code implementations LREC 2016 Emiel van Miltenburg, Benjamin Timmermans, Lora Aroyo

The main goal of this study is to find out (i) whether it is feasible to collect keywords for a large collection of sounds through crowdsourcing, and (ii) how people talk about sounds, and what information they can infer from hearing a sound in isolation.

Cannot find the paper you are looking for? You can Submit a new open access paper.