TVStoryGen: A Dataset for Generating Stories with Character Descriptions

18 Sep 2021  ยท  Mingda Chen, Kevin Gimpel ยท

We introduce TVStoryGen, a story generation dataset that requires generating detailed TV show episode recaps from a brief summary and a set of documents describing the characters involved. Unlike other story generation datasets, TVStoryGen contains stories that are authored by professional screen-writers and that feature complex interactions among multiple characters. Generating stories in TVStoryGen requires drawing relevant information from the lengthy provided documents about characters based on the brief summary. In addition, we propose to train reverse models on our dataset for evaluating the faithfulness of generated stories. We create TVStoryGen from fan-contributed websites, which allows us to collect 26k episode recaps with 1868.7 tokens on average. Empirically, we take a hierarchical story generation approach and find that the neural model that uses oracle content selectors for character descriptions demonstrates the best performance on automatic metrics, showing the potential of our dataset to inspire future research on story generation with constraints. Qualitative analysis shows that the best-performing model sometimes generates content that is unfaithful to the short summaries, suggesting promising directions for future work.

PDF Abstract

Datasets


Introduced in the Paper:

TVRecap

Used in the Paper:

ROCStories WritingPrompts

Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Story Generation Fandom dev (NN) Oracle plot + summary + oracle char. desc. BLEU 28.4 # 1
ROUGE-1 63.0 # 1
ROUGE-2 32.8 # 1
ROUGE-L 61.2 # 1
Perplexity 17.9 # 1
Story Generation Fandom test (NN) Oracle plot + summary + oracle char. desc. BLEU 28.4 # 1
ROUGE-1 63.2 # 1
ROUGE-2 32.9 # 1
ROUGE-L 61.5 # 1
Perplexity 18.2 # 1
Story Generation TVMegaSite dev (NN) Oracle plot + summary + oracle char. desc. BLEU 30.9 # 1
ROUGE-1 68.3 # 1
ROUGE-2 44.0 # 1
ROUGE-L 67.5 # 1
Perplexity 15.7 # 1
Story Generation TVMegaSite test (NN) Oracle plot + summary + oracle char. desc. BLEU 28.1 # 1
ROUGE-1 67.0 # 1
ROUGE-2 40.9 # 1
Perplexity 18.3 # 1
ROUGE-L 66.2 # 1

Methods


No methods listed for this paper. Add relevant methods here