Search Results for author:
Found 38 papers, 15 papers with code
longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks.
Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability.
Benchmark Transparency: Measuring the Impact of Data on Evaluation
In this paper we present an exploratory research on quantifying the impact that data distribution has on the performance and evaluation of NLP models.
Diverse, but Divisive: LLMs Can Exaggerate Gender Differences in Opinion Related to Harms of Misinformation
We pose two central questions: (1) To what extent do prompts with explicit gender references reflect gender differences in opinion in the United States on topics of social relevance?
A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks
Beyond investigating these research questions above, we discuss the foundational concept of annotation complexity, present a new aggregation model as a bridge between traditional models and our own, and contribute a new semi-supervised learning method for complex label aggregation that outperforms prior work.
DMLR: Data-centric Machine Learning Research -- Past, Present and Future
Drawing from discussions at the inaugural DMLR workshop at ICML 2023 and meetings prior, in this report we outline the relevance of community engagement and infrastructure development for the creation of next-generation public datasets that will advance machine learning science.
Interpretable by Design: Wrapper Boxes Combine Neural Performance with Faithful Explanations
Can we preserve the accuracy of neural models while also providing faithful explanations?
Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI
While many Natural Language Processing (NLP) techniques have been proposed for fact-checking, both academic research and fact-checking organizations report limited adoption of such NLP work due to poor alignment with fact-checker practices, values, and needs.
Designing Closed-Loop Models for Task Allocation
Instead, we find ourselves in a "closed" decision-making loop in which the same fallible human decisions we rely on in practice must also be used to guide task allocation.
Same Same, But Different: Conditional Multi-Task Learning for Demographic-Specific Toxicity Detection
We propose framing toxicity detection as multi-task learning (MTL), allowing a model to specialize on the relationships that are relevant to each demographic group while also leveraging shared properties across groups.
Learning Complementary Policies for Human-AI Teams
We then extend our approach to leverage opportunities and mitigate risks that arise in important contexts in practice: 1) when a team is composed of multiple humans with differential and potentially complementary abilities, 2) when the observational data includes consistent deterministic actions, and 3) when the covariate distribution of future decisions differ from that in the historical data.
New Metrics to Encourage Innovation and Diversity in Information Retrieval Approaches
In addition, we show that our metrics achieve higher evaluation stability and discriminative power than the standard metrics we modify.
The State of Human-centered NLP Technology for Fact-checking
We recommend that future research include collaboration with fact-checker stakeholders early on in NLP research, as well as incorporation of human-centered design practices in model development, in order to further guide technology development for human use and practical adoption.
Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks
When annotators label data, a key metric for quality assurance is inter-annotator agreement (IAA): the extent to which annotators agree on their labels.
longhorns at DADC 2022: How many linguists does it take to fool a Question Answering model? A systematic approach to adversarial attacks
Developing methods to adversarially challenge NLP systems is a promising avenue for improving both model performance and interpretability.
Fairly Accurate: Learning Optimal Accuracy vs. Fairness Tradeoffs for Hate Speech Detection
In this work, we first introduce a differentiable measure that enables direct optimization of group fairness (specifically, balancing accuracy across groups) in model training.
ProtoTEx: Explaining Model Decisions with Prototype Tensors
We present ProtoTEx, a novel white-box NLP classification architecture based on prototype networks.
The Effects of Interactive AI Design on User Behavior: An Eye-tracking Study of Fact-checking COVID-19 Claims
We conducted a lab-based eye-tracking study to investigate how the interactivity of an AI-powered fact-checking system affects user interactions, such as dwell time, attention, and mental resources involved in using the system.
Designing Closed Human-in-the-loop Deferral Pipelines
Our key insight is that by exploiting weak prior information, we can match experts to input examples to ensure fairness and accuracy of the resulting deferral framework, even when imperfect and biased experts are used in place of ground truth labels.
In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd Workers
We propose a novel three-stage FIND-RESOLVE-LABEL workflow for crowdsourced annotation to reduce ambiguity in task instructions and thus improve annotation quality.
Data Excellence for AI: Why Should You Care
The efficacy of machine learning (ML) models depends on both algorithms and data.
Scalable Unidirectional Pareto Optimality for Multi-Task Learning with Constraints
Multi-objective optimization (MOO) problems require balancing competing objectives, often under constraints.
A Two-Stage Neural-Filter Pareto Front Extractor and the need for Benchmarking
The Stage-1 neural network efficiently extracts the \textit{weak} Pareto front, using Fritz-John Conditions (FJC) as the discriminator, with no assumptions of convexity on the objectives or constraints.
The Case for Claim Difficulty Assessment in Automatic Fact Checking
Fact-checking is the process of evaluating the veracity of claims (i. e., purported facts).
An Information Retrieval Approach to Building Datasets for Hate Speech Detection
Our key insight is that the rarity and subjectivity of hate speech are akin to that of relevance in information retrieval (IR).
Towards Unbiased and Accurate Deferral to Multiple Experts
Machine learning models are often implemented in cohort with humans in the pipeline, with the model having an option to defer to a domain expert in cases where it has low confidence in its inference.
A Hybrid 2-stage Neural Optimization for Pareto Front Extraction
The first stage (neural network) efficiently extracts a weak Pareto front, using Fritz-John conditions as the discriminator, with no assumptions of convexity on the objectives or constraints.
Understanding and Predicting Characteristics of Test Collections in Information Retrieval
Research community evaluations in information retrieval, such as NIST's Text REtrieval Conference (TREC), build reusable test collections by pooling document rankings submitted by many teams.
You Are What You Tweet: Profiling Users by Past Tweets to Improve Hate Speech Detection
Hate speech detection research has predominantly focused on purely content-based methods, without exploiting any additional context.
Range-Net: A High Precision Streaming SVD for Big Data Applications
Although these methods are claimed to be applicable to scientific computations due to associated tail-energy error bounds, the approximation errors in the singular vectors and values are high when the aforementioned assumption does not hold.
Extracting Optimal Solution Manifolds using Constrained Neural Optimization
However, such an approach is often restricted to a strict class of functions, deviation from which results in sub-optimal solution to the original problem.
A Conceptual Framework for Evaluating Fairness in Search
While search efficacy has been evaluated traditionally on the basis of result relevance, fairness of search has attracted recent attention.
CobWeb: A Research Prototype for Exploring User Bias in Political Fact-Checking
The effect of user bias in fact-checking has not been explored extensively from a user-experience perspective.
Efficient Test Collection Construction via Active Learning
To create a new IR test collection at low cost, it is valuable to carefully select which documents merit human relevance judgments.
Aggregating and Predicting Sequence Labels from Crowd Annotations
Despite sequences being core to NLP, scant work has considered how to handle noisy sequence labels from multiple annotators for the same text.
Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization
A fundamental advantage of neural models for NLP is their ability to learn representations from scratch.
Neural Information Retrieval: A Literature Review
A recent "third wave" of Neural Network (NN) approaches now delivers state-of-the-art performance in many machine learning tasks, spanning speech recognition, computer vision, and natural language processing.
Active Discriminative Text Representation Learning
We also show that, as expected, the method quickly learns discriminative word embeddings.
TurKPF: TurKontrol as a Particle Filter
TurKontrol, and algorithm presented in (Dai et al. 2010), uses a POMDP to model and control an iterative workflow for crowdsourced work.
