Search Results for author:
Found 37 papers, 6 papers with code
Disentangling the Properties of Human Evaluation Methods: A Classification System to Support Comparability, Meta-Evaluation and Reproducibility Testing
Current standards for designing and reporting human evaluations in NLP mean it is generally unclear which evaluations are comparable and can be expected to yield similar results when applied to the same system outputs.
The 2020 Bilingual, Bi-Directional WebNLG+ Shared Task: Overview and Evaluation Results (WebNLG+ 2020)
WebNLG+ offers two challenges: (i) mapping sets of RDF triples to English or Russian text (generation) and (ii) converting English or Russian text to sets of RDF triples (semantic parsing).
A General Benchmarking Framework for Text Generation
The RDF-to-text task has recently gained substantial attention due to the continuous growth of RDF knowledge graphs in number and size.
A Case Study of NLG from Multimedia Data Sources: Generating Architectural Landmark Descriptions
In this paper, we present a pipeline system that generates architectural landmark descriptions using textual, visual and structured data.
Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions
Human assessment remains the most trusted form of evaluation in NLG, but highly diverse approaches and a proliferation of different quality criteria used by researchers make it difficult to compare results and draw conclusions across papers, with adverse implications for meta-evaluation and reproducibility.
Another PASS: A Reproduction Study of the Human Evaluation of a Football Report Generation System
Clarity had a higher degree of reproducibility than Fluency, as measured by the coefficient of variation.
Text-in-Context: Token-Level Error Detection for Table-to-Text Generation
Our system can detect the errors automatically using a combination of a rule-based natural language generation (NLG) system and pretrained language models (LMs).
The Third Multilingual Surface Realisation Shared Task (SR’20): Overview and Evaluation Results
As in SR’18 and SR’19, the shared task comprised two tracks: (1) a Shallow Track where the inputs were full UD structures with word order information removed and tokens lemmatised; and (2) a Deep Track where additionally, functional words and morphological information were removed.
On the Role of Summary Content Units in Text Summarization Evaluation
At the heart of the Pyramid evaluation method for text summarization lie human written summary content units (SCUs).
Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization
To prevent the costly and inefficient use of resources on low-quality annotations, we want a method for creating a pool of dependable annotators who can effectively complete difficult tasks, such as evaluating automatic summarization.
GEMv2: Multilingual NLG Benchmarking in a Single Line of Code
This problem is especially pertinent in natural language generation which requires ever-improving suites of datasets, metrics, and human evaluation to make definitive claims.
Quantified Reproducibility Assessment of NLP Results
This paper describes and tests a method for carrying out quantified reproducibility assessment (QRA) that is based on concepts and definitions from metrology.
NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentation
Data augmentation is an important component in the robustness evaluation of models in natural language processing (NLP) and in enhancing the diversity of the data they are trained on.
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
By applying this framework to the GEM generation benchmark, we propose an evaluation suite made of 80 challenge sets, demonstrate the kinds of analyses that it enables and shed light onto the limits of current generation models.
Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models
More specifically, we evaluate the syntactic generalization potential of the models on English and Spanish tests, comparing the syntactic abilities of monolingual and multilingual models on the same language (English), and of multilingual models on two different languages (English and Spanish).
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
We introduce GEM, a living benchmark for natural language Generation (NLG), its Evaluation, and Metrics.
Ranked #1 on
Extreme Summarization
on GEM-XSum
Abstractive Text Summarization
Cross-Lingual Abstractive Summarization
+5
Back-Translation as Strategy to Tackle the Lack of Corpus in Natural Language Generation from Semantic Representations
This paper presents an exploratory study that aims to evaluate the usefulness of back-translation in Natural Language Generation (NLG) from semantic representations for non-English languages.
The Second Multilingual Surface Realisation Shared Task (SR'19): Overview and Evaluation Results
We report results from the SR{'}19 Shared Task, the second edition of a multilingual surface realisation task organised as part of the EMNLP{'}19 Workshop on Multilingual Surface Realisation.
Teaching FORGe to Verbalize DBpedia Properties in Spanish
Statistical generators increasingly dominate the research in NLG.
Sentence Packaging in Text Generation from Semantic Graphs as a Community Detection Problem
An increasing amount of research tackles the challenge of text generation from abstract ontological or semantic structures, which are in their very nature potentially large connected graphs.
Underspecified Universal Dependency Structures as Inputs for Multilingual Surface Realisation
In this paper, we present the datasets used in the Shallow and Deep Tracks of the First Multilingual Surface Realisation Shared Task (SR{'}18).
The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results
We report results from the SR{'}18 Shared Task, a new multilingual surface realisation task organised as part of the ACL{'}18 Workshop on Multilingual Surface Realisation.
Shared Task Proposal: Multilingual Surface Realization Using Universal Dependency Trees
We propose a shared task on multilingual Surface Realization, i. e., on mapping unordered and uninflected universal dependency trees to correctly ordered and inflected sentences in a number of languages.
A demo of FORGe: the Pompeu Fabra Open Rule-based Generator
This demo paper presents the multilingual deep sentence generator developed by the TALN group at Universitat Pompeu Fabra, implemented as a series of rule-based graph-transducers for the syntacticization of the input graphs, the resolution of morphological agreements, and the linearization of the trees.
FORGe at SemEval-2017 Task 9: Deep sentence generation based on a sequence of graph transducers
We present the contribution of Universitat Pompeu Fabra{'}s NLP group to the SemEval Task 9. 2 (AMR-to-English Generation).
Text Simplification Tools for Spanish
In this paper we describe the development of a text simplification system for Spanish.
