Search Results for author: Lora Aroyo

Found 25 papers, 13 papers with code

Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection

1 code implementation22 Aug 2023 Oana Inel, Tim Draws, Lora Aroyo

We argue that data collection for AI should be performed in a responsible manner where the quality of the data is thoroughly scrutinized and measured through a systematic set of appropriate metrics.


"Is a picture of a bird a bird": Policy recommendations for dealing with ambiguity in machine vision models

no code implementations27 Jun 2023 Alicia Parrish, Sarah Laszlo, Lora Aroyo

Many questions that we ask about the world do not have a single clear answer, yet typical human annotation set-ups in machine learning assume there must be a single ground truth label for all examples in every task.

Measuring Attribution in Natural Language Generation Models

1 code implementation23 Dec 2021 Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, David Reitter

With recent improvements in natural language generation (NLG) models for various applications, it has become imperative to have the means to identify and evaluate whether NLG output is only sharing verifiable information about the external world.

Text Generation

Data Excellence for AI: Why Should You Care

no code implementations19 Nov 2021 Lora Aroyo, Matthew Lease, Praveen Paritosh, Mike Schaekermann

The efficacy of machine learning (ML) models depends on both algorithms and data.

Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability

no code implementations ACL 2021 Ka Wong, Praveen Paritosh, Lora Aroyo

When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012).


Metrology for AI: From Benchmarks to Instruments

1 code implementation5 Nov 2019 Chris Welty, Praveen Paritosh, Lora Aroyo

In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations.

Word Similarity

A Crowdsourced Frame Disambiguation Corpus with Ambiguity

1 code implementation NAACL 2019 Anca Dumitrache, Lora Aroyo, Chris Welty

We present a resource for the task of FrameNet semantic frame disambiguation of over 5, 000 word-sentence pairs from the Wikipedia corpus.


Homonym Detection For Humor Recognition In Short Text

no code implementations WS 2018 Sven van den Beukel, Lora Aroyo

In this paper, automatic homophone- and homograph detection are suggested as new useful features for humor recognition systems.

General Classification

Crowdsourcing Semantic Label Propagation in Relation Classification

1 code implementation WS 2018 Anca Dumitrache, Lora Aroyo, Chris Welty

Distant supervision is a popular method for performing relation extraction from text that is known to produce noisy labels.

Classification General Classification +1

CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement

2 code implementations18 Aug 2018 Anca Dumitrache, Oana Inel, Lora Aroyo, Benjamin Timmermans, Chris Welty

However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples.

Human-Computer Interaction Social and Information Networks

Capturing Ambiguity in Crowdsourcing Frame Disambiguation

1 code implementation1 May 2018 Anca Dumitrache, Lora Aroyo, Chris Welty

FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words.

False Positive and Cross-relation Signals in Distant Supervision Data

1 code implementation14 Nov 2017 Anca Dumitrache, Lora Aroyo, Chris Welty

Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair, then sentences that contain that pair are likely to express the relation.

General Classification Relation Classification

Crowdsourcing Ground Truth for Medical Relation Extraction

1 code implementation9 Jan 2017 Anca Dumitrache, Lora Aroyo, Chris Welty

Cognitive computing systems require human labeled data for evaluation, and often for training.

Medical Relation Extraction Relation Extraction

Crowdsourcing Salient Information from News and Tweets

no code implementations LREC 2016 Oana Inel, Tommaso Caselli, Lora Aroyo

On the other hand, machines need to understand the information that is published in online data streams and generate concise and meaningful overviews.

The VU Sound Corpus: Adding More Fine-grained Annotations to the Freesound Database

no code implementations LREC 2016 Emiel van Miltenburg, Benjamin Timmermans, Lora Aroyo

The main goal of this study is to find out (i) whether it is feasible to collect keywords for a large collection of sounds through crowdsourcing, and (ii) how people talk about sounds, and what information they can infer from hearing a sound in isolation.

GRaSP: A Multilayered Annotation Scheme for Perspectives

no code implementations LREC 2016 Chantal van Son, Tommaso Caselli, Antske Fokkens, Isa Maks, Roser Morante, Lora Aroyo, Piek Vossen

In the last decade, different aspects of linguistic encoding of perspectives have been targeted as separated phenomena through different annotation initiatives.

Cannot find the paper you are looking for? You can Submit a new open access paper.