Hierarchically-Attentive RNN for Album Summarization and Storytelling

EMNLP 2017  ·  Licheng Yu, Mohit Bansal, Tamara L. Berg ·

We address the problem of end-to-end visual storytelling. Given a photo album, our model first selects the most representative (summary) photos, and then composes a natural language story for the album. For this task, we make use of the Visual Storytelling dataset and a model composed of three hierarchically-attentive Recurrent Neural Nets (RNNs) to: encode the album photos, select representative (summary) photos, and compose the story. Automatic and human evaluations show our model achieves better performance on selection, generation, and retrieval than baselines.

PDF Abstract EMNLP 2017 PDF EMNLP 2017 Abstract

Datasets


Results from the Paper


Ranked #15 on Visual Storytelling on VIST (BLEU-3 metric)

     Get a GitHub badge
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Visual Storytelling VIST h-attn-rank BLEU-3 20.78 # 15
METEOR 33.94 # 27
CIDEr 7.38 # 24
ROUGE-L 29.82 # 18

Methods


No methods listed for this paper. Add relevant methods here