Search Results for author:
Found 12 papers, 3 papers with code
LINGO : Visually Debiasing Natural Language Instructions to Support Task Diversity
Cross-task generalization is a significant outcome that defines mastery in natural language understanding.
Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow
In pursuit of creating better benchmarks, we propose VAIDA, a novel benchmark creation paradigm for NLP, that focuses on guiding crowdworkers, an under-explored facet of addressing benchmark idiosyncrasies.
Hardness of Samples Need to be Quantified for a Reliable Evaluation System: Exploring Potential Opportunities with a New Task
Evaluation of models on benchmarks is unreliable without knowing the degree of sample hardness; this subsequently overestimates the capability of AI systems and limits their adoption in real world applications.
A Survey of Parameters Associated with the Quality of Benchmarks in NLP
Inspired by successful quality indices in several domains such as power, food, and water, we take the first step towards a metric by identifying certain language properties that can represent various possible interactions leading to biases in a benchmark.
Investigating the Failure Modes of the AUC metric and Exploring Alternatives for Evaluating Systems in Safety Critical Applications
We find limitations in AUC; e. g., a model having higher AUC is not always better in performing selective answering.
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions -- training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.
A Proposal to Study "Is High Quality Data All We Need?"
Our hypothesis is based on the fact that deep neural networks are data driven models, and data is what leads/misleads models.
How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation
Models that top leaderboards often perform unsatisfactorily when deployed in real world applications; this has necessitated rigorous and expensive pre-deployment model testing.
Front Contribution instead of Back Propagation
We show that our algorithm produces the exact same output as BP, in contrast to several recently proposed algorithms approximating BP.
DQI: A Guide to Benchmark Evaluation
A `state of the art' model A surpasses humans in a benchmark B, but fails on similar benchmarks C, D, and E. What does B have that the other benchmarks do not?
Our Evaluation Metric Needs an Update to Encourage Generalization
In order to stop the inflation in model performance -- and thus overestimation in AI systems' capabilities -- we propose a simple and novel evaluation metric, WOOD Score, that encourages generalization during evaluation.
DQI: Measuring Data Quality in NLP
The data creation paradigm consists of several data visualizations to help data creators (i) understand the quality of data and (ii) visualize the impact of the created data instance on the overall quality.
