Search Results for author:
Found 15 papers, 4 papers with code
Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods.
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?
While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models).
py-irt: A Scalable Item Response Theory Library for Python
py-irt is a Python library for fitting Bayesian Item Response Theory (IRT) models.
Constructing a Psychometric Testbed for Fair Natural Language Processing
We also discuss the important implications of our work and resulting testbed for future NLP research on psychometrics and fairness.
Understanding Deep Learning Performance through an Examination of Test Set Difficulty: A Psychometric Case Study
We examine the impact of a test set question's difficulty to determine if there is a relationship between difficulty and performance.
Soft Label Memorization-Generalization for Natural Language Inference
Often when multiple labels are obtained for a training example it is assumed that there is an element of noise that must be accounted for.
Building an Evaluation Scale using Item Response Theory
Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1).
Efficient Semi-Supervised Learning for Natural Language Understanding by Optimizing Diversity
In addition, methods optimizing diversity can reduce training data in many cases to 50% with little impact on performance.
Natural Language Understanding
Task-Oriented Dialogue Systems
Dynamic Data Selection for Curriculum Learning via Ability Estimation
Curriculum learning methods typically rely on heuristics to estimate the difficulty of training examples or the ability of the model.
An Empirical Analysis of Human-Bot Interaction on Reddit
Automated agents (“bots”) have emerged as an ubiquitous and influential presence on social media.
Measuring algorithmic interpretability: A human-learning-based framework and the corresponding cognitive complexity score
We illustrate the measurement framework through a toy example, describe the framework and its conceptual underpinnings, and demonstrate the benefits of the framework, in particular for managers considering tradeoffs when selecting algorithms.
Stars Are All You Need: A Distantly Supervised Pyramid Network for Unified Sentiment Analysis
We also demonstrate the interpretability of DSPN's outputs on reviews to show the pyramid structure inherent in unified sentiment analysis.
Aspect Category Detection
Aspect Category Sentiment Analysis
+1
Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads
Although a burgeoning literature has emerged on stereotypical bias mitigation in PLMs, such as work on debiasing gender and racial stereotyping, how such biases manifest and behave internally within PLMs remains largely unknown.
H-COAL: Human Correction of AI-Generated Labels for Biomedical Named Entity Recognition
With the rapid advancement of machine learning models for NLP tasks, collecting high-fidelity labels from AI models is a realistic possibility.
