Search Results for author:
Found 6 papers, 1 papers with code
Proving membership in LLM pretraining data via data watermarks
Detecting whether copyright holders' works were used in LLM pretraining is poised to be an important problem.
Searching for a higher power in the human evaluation of MT
In MT evaluation, pairwise comparisons are conducted to identify the better system.
The statistical advantage of automatic NLG metrics at the system level
Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected.
On conducting better validation studies of automatic metrics in natural language generation evaluation
This document is intended for those validating existing metrics or proposing new ones in the broad context of NLG: we 1) begin with a write-up of best practices in validation studies, 2) outline how to adopt these practices, 3) conduct analyses in the WMT'17 metrics shared task\footnote{Our jupyter notebook containing the analyses is available at \url{https://github. com}}, and 4) highlight promising approaches to NLG metrics 5) conclude with our opinions on the future of this area.
Better Automatic Evaluation of Open-Domain Dialogue Systems with Contextualized Embeddings
Despite advances in open-domain dialogue systems, automatic evaluation of such systems is still a challenging problem.
Evaluating Syntactic Properties of Seq2seq Output with a Broad Coverage HPSG: A Case Study on Machine Translation
We explore whether such output belongs to a formal and realistic grammar, by employing the English Resource Grammar (ERG), a broad coverage, linguistically precise HPSG-based grammar of English.
