Search Results for author:
Found 15 papers, 9 papers with code
RESIN-11: Schema-guided Event Prediction for 11 Newsworthy Scenarios
We introduce RESIN-11, a new schema-guided event extraction&prediction framework that can be applied to a large variety of newsworthy scenarios.
State of What Art? A Call for Multi-Prompt LLM Evaluation
Recent advances in large language models (LLMs) have led to the development of various evaluation benchmarks.
DMLR: Data-centric Machine Learning Research -- Past, Present and Future
Drawing from discussions at the inaugural DMLR workshop at ICML 2023 and meetings prior, in this report we outline the relevance of community engagement and infrastructure development for the creation of next-generation public datasets that will advance machine learning science.
The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Specifically, we propose a novel competition setting in which we select a list of allowed LLMs and disallow fine-tuning to ensure a focus on prompting.
Human-in-the-Loop Schema Induction
Schema induction builds a graph representation explaining how events unfold in a scenario.
On the Limitations of Reference-Free Evaluations of Generated Text
There is significant interest in developing evaluation metrics which accurately estimate the quality of generated text without the aid of a human-written reference text, which can be time consuming and expensive to collect or entirely unavailable in online applications.
Zero-Shot On-the-Fly Event Schema Induction
The answers to these questions can be found by collecting many documents on the complex event of interest, extracting relevant information, and analyzing it.
Re-Examining System-Level Correlations of Automatic Summarization Evaluation Metrics
How reliably an automatic summarization evaluation metric replicates human judgments of summary quality is quantified by system-level correlations.
A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods
After evaluating which of the proposed methods is most appropriate for summarization through two simulation experiments, we analyze the results of applying these methods to several different automatic evaluation metrics across three sets of human annotations.
The Structured Weighted Violations MIRA
We present the Structured Weighted Violation MIRA (SWVM), a new structured prediction algorithm that is based on an hybridization between MIRA (Crammer and Singer, 2003) and the structured weighted violations perceptron (SWVP) (Dror and Reichart, 2016).
Deep Dominance - How to Properly Compare Deep Neural Models
Comparing between Deep Neural Network (DNN) models based on their performance on unseen data is crucial for the progress of the NLP field.
Appendix - Recommended Statistical Significance Tests for NLP Tasks
Statistical significance testing plays an important role when drawing conclusions from experimental results in NLP papers.
The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing
We establish the fundamental concepts of significance testing and discuss the specific aspects of NLP tasks, experimental setups and evaluation measures that affect the choice of significance tests in NLP research.
Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets
With the ever-growing amounts of textual data from a large variety of languages, domains, and genres, it has become standard to evaluate NLP algorithms on multiple datasets in order to ensure consistent performance across heterogeneous setups.
The Structured Weighted Violations Perceptron Algorithm
We present the Structured Weighted Violations Perceptron (SWVP) algorithm, a new structured prediction algorithm that generalizes the Collins Structured Perceptron (CSP).
