Search Results for author: John P. Lalor

Found 12 papers, 4 papers with code

An Empirical Analysis of Human-Bot Interaction on Reddit

no code implementations EMNLP (WNUT) 2020 Ming-Cheng Ma, John P. Lalor

Automated agents (“bots”) have emerged as an ubiquitous and influential presence on social media.

Measuring algorithmic interpretability: A human-learning-based framework and the corresponding cognitive complexity score

no code implementations20 May 2022 John P. Lalor, Hong Guo

We illustrate the measurement framework through a toy example, describe the framework and its conceptual underpinnings, and demonstrate the benefits of the framework, in particular for managers considering tradeoffs when selecting algorithms.

Fairness

py-irt: A Scalable Item Response Theory Library for Python

1 code implementation2 Mar 2022 John P. Lalor, Pedro Rodriguez

py-irt is a Python library for fitting Bayesian Item Response Theory (IRT) models.

Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards?

1 code implementation ACL 2021 Pedro Rodriguez, Joe Barrow, Alexander Miserlis Hoyle, John P. Lalor, Robin Jia, Jordan Boyd-Graber

While leaderboards are a straightforward ranking of NLP models, this simplicity can mask nuances in evaluation items (examples) and subjects (NLP models).

Dynamic Data Selection for Curriculum Learning via Ability Estimation

no code implementations Findings of the Association for Computational Linguistics 2020 John P. Lalor, Hong Yu

Curriculum learning methods typically rely on heuristics to estimate the difficulty of training examples or the ability of the model.

Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds

1 code implementation IJCNLP 2019 John P. Lalor, Hao Wu, Hong Yu

We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods.

Natural Language Inference Sentiment Analysis

Soft Label Memorization-Generalization for Natural Language Inference

no code implementations27 Feb 2017 John P. Lalor, Hao Wu, Hong Yu

Often when multiple labels are obtained for a training example it is assumed that there is an element of noise that must be accounted for.

Natural Language Inference

Building an Evaluation Scale using Item Response Theory

no code implementations EMNLP 2016 John P. Lalor, Hao Wu, Hong Yu

Evaluation of NLP methods requires testing against a previously vetted gold-standard test set and reporting standard metrics (accuracy/precision/recall/F1).

Natural Language Inference

Cannot find the paper you are looking for? You can Submit a new open access paper.