Search Results for author:
Found 25 papers, 13 papers with code
Annotator Response Distributions as a Sampling Frame
Annotator disagreement is often dismissed as noise or the result of poor annotation process quality.
Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection
We argue that data collection for AI should be performed in a responsible manner where the quality of the data is thoroughly scrutinized and measured through a systematic set of appropriate metrics.
"Is a picture of a bird a bird": Policy recommendations for dealing with ambiguity in machine vision models
Many questions that we ask about the world do not have a single clear answer, yet typical human annotation set-ups in machine learning assume there must be a single ground truth label for all examples in every task.
Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models
To address this need, we introduce the Adversarial Nibbler challenge.
Human-Centered Responsible Artificial Intelligence: Current & Future Trends
In recent years, the CHI community has seen significant growth in research on Human-Centered Responsible Artificial Intelligence.
DataPerf: Benchmarks for Data-Centric AI Development
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems.
LaMDA: Language Models for Dialog Applications
We demonstrate that fine-tuning with annotated data and enabling the model to consult external knowledge sources can lead to significant improvements towards the two key challenges of safety and factual grounding.
Ranked #40 on
Code Generation
on HumanEval
Measuring Attribution in Natural Language Generation Models
With recent improvements in natural language generation (NLG) models for various applications, it has become imperative to have the means to identify and evaluate whether NLG output is only sharing verifiable information about the external world.
Data Excellence for AI: Why Should You Care
The efficacy of machine learning (ML) models depends on both algorithms and data.
Cross-replication Reliability - An Empirical Approach to Interpreting Inter-rater Reliability
When collecting annotations and labeled data from humans, a standard practice is to use inter-rater reliability (IRR) as a measure of data goodness (Hallgren, 2012).
Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability
We present a new approach to interpreting IRR that is empirical and contextualized.
Metrology for AI: From Benchmarks to Instruments
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying it to human (crowd) powered evaluations.
A Crowdsourced Frame Disambiguation Corpus with Ambiguity
We present a resource for the task of FrameNet semantic frame disambiguation of over 5, 000 word-sentence pairs from the Wikipedia corpus.
Homonym Detection For Humor Recognition In Short Text
In this paper, automatic homophone- and homograph detection are suggested as new useful features for humor recognition systems.
Crowdsourcing Semantic Label Propagation in Relation Classification
Distant supervision is a popular method for performing relation extraction from text that is known to produce noisy labels.
CrowdTruth 2.0: Quality Metrics for Crowdsourcing with Disagreement
However, in many domains, there is ambiguity in the data, as well as a multitude of perspectives of the information examples.
Human-Computer Interaction Social and Information Networks
Scoring and Classifying Implicit Positive Interpretations: A Challenge of Class Imbalance
This paper reports on a reimplementation of a system on detecting implicit positive meaning from negated statements.
Capturing Ambiguity in Crowdsourcing Frame Disambiguation
FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words.
False Positive and Cross-relation Signals in Distant Supervision Data
Distant supervision (DS) is a well-established method for relation extraction from text, based on the assumption that when a knowledge-base contains a relation between a term pair, then sentences that contain that pair are likely to express the relation.
Crowdsourcing Ground Truth for Medical Relation Extraction
Cognitive computing systems require human labeled data for evaluation, and often for training.
Crowdsourcing Salient Information from News and Tweets
On the other hand, machines need to understand the information that is published in online data streams and generate concise and meaningful overviews.
The VU Sound Corpus: Adding More Fine-grained Annotations to the Freesound Database
The main goal of this study is to find out (i) whether it is feasible to collect keywords for a large collection of sounds through crowdsourcing, and (ii) how people talk about sounds, and what information they can infer from hearing a sound in isolation.
GRaSP: A Multilayered Annotation Scheme for Perspectives
In the last decade, different aspects of linguistic encoding of perspectives have been targeted as separated phenomena through different annotation initiatives.
