Coherence Evaluation
15 papers with code • 2 benchmarks • 1 datasets
Evaluating the overall coherence of text as measured by its readability and flow through ideas.
Most implemented papers
Transformer Models for Text Coherence Assessment
Coherence is an important aspect of text quality and is crucial for ensuring its readability.
Discourse Coherence in the Wild: A Dataset, Evaluation and Methods
To date there has been very little work on assessing discourse coherence methods on real-world data.
Neural RST-based Evaluation of Discourse Coherence
We evaluate our approach on the Grammarly Corpus for Discourse Coherence (GCDC) and show that when ensembled with the current state of the art, we can achieve the new state of the art accuracy on this benchmark.
Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models
Coherent discourse is distinguished from a mere collection of utterances by the satisfaction of a diverse set of constraints, for example choice of expression, logical relation between denoted events, and implicit compatibility with world-knowledge.
Towards Quantifiable Dialogue Coherence Evaluation
To address these limitations, we propose Quantifiable Dialogue Coherence Evaluation (QuantiDCE), a novel framework aiming to train a quantifiable dialogue coherence metric that can reflect the actual human rating standards.
DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulations
We also show that DEAM can distinguish between coherent and incoherent dialogues generated by baseline manipulations, whereas those baseline models cannot detect incoherent examples generated by DEAM.
SNaC: Coherence Error Detection for Narrative Summarization
In this work, we introduce SNaC, a narrative coherence evaluation framework rooted in fine-grained annotations for long summaries.
GisPy: A Tool for Measuring Gist Inference Score in Text
Decision making theories such as Fuzzy-Trace Theory (FTT) suggest that individuals tend to rely on gist, or bottom-line meaning, in the text when making decisions.
Transform, Contrast and Tell: Coherent Entity-Aware Multi-Image Captioning
Experiments also show that the generated captions are more coherent than that of baselines according to caption entity scores, caption Rouge scores, the two proposed coherence evaluation metrics, and human evaluations.
Open-Domain Text Evaluation via Contrastive Distribution Methods
We investigate CDM for open-domain text generation evaluation under two paradigms: 1) _Generative_ CDM, which harnesses the contrast of two language models' distributions to generate synthetic examples for training discriminator-based metrics; 2) _Discriminative_ CDM, which directly uses distribution disparities between two language models for evaluation.