HowSumm: A Multi-Document Summarization Dataset Derived from WikiHow Articles

7 Oct 2021  ยท  Odellia Boni, Guy Feigenblat, Guy Lev, Michal Shmueli-Scheuer, Benjamin Sznajder, David Konopnicki ยท

We present HowSumm, a novel large-scale dataset for the task of query-focused multi-document summarization (qMDS), which targets the use-case of generating actionable instructions from a set of sources. This use-case is different from the use-cases covered in existing multi-document summarization (MDS) datasets and is applicable to educational and industrial scenarios. We employed automatic methods, and leveraged statistics from existing human-crafted qMDS datasets, to create HowSumm from wikiHow website articles and the sources they cite. We describe the creation of the dataset and discuss the unique features that distinguish it from other summarization corpora. Automatic and human evaluations of both extractive and abstractive summarization models on the dataset reveal that there is room for improvement.

PDF Abstract

Datasets


Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Document Summarization HowSumm-Method LexRank (query: method + article + steps titles) ROUGE-1 53.5 # 1
Document Summarization HowSumm-Method GreedyRel (query: method + article + steps titles) ROUGE-1 48.6 # 3
Document Summarization HowSumm-Method CES (query: method + article titles) ROUGE-1 48.3 # 5
Document Summarization HowSumm-Method LexRank (query: method + article titles) ROUGE-1 47.1 # 7
Document Summarization HowSumm-Method GreedyRel (query: method + article titles) ROUGE-1 42.3 # 9
Document Summarization HowSumm-Method CES (query: method title) ROUGE-1 48.4 # 4
Document Summarization HowSumm-Method LexRank (query: method title) ROUGE-1 47.7 # 6
Document Summarization HowSumm-Method GreedyRel (query: method title) ROUGE-1 43.4 # 8
Document Summarization HowSumm-Method CES (query: method + article + steps titles) ROUGE-1 52.2 # 2
Document Summarization HowSumm-Step CES (query: step title) ROUGE-1 39.3 # 2
Document Summarization HowSumm-Step LexRank (query: step title) ROUGE-1 39.6 # 1
Document Summarization HowSumm-Step GreedyRel (query: step title) ROUGE-1 30.1 # 8
Document Summarization HowSumm-Step BM25-HierSumm (query: step + method + article titles) ROUGE-1 21.9 # 11
Document Summarization HowSumm-Step CES (query: step + method + article titles) ROUGE-1 37.0 # 5
Document Summarization HowSumm-Step LexRank (query: step + method + article titles) ROUGE-1 36.3 # 6
Document Summarization HowSumm-Step BM25-HierSumm (query: step + method titles) ROUGE-1 23.0 # 9
Document Summarization HowSumm-Step CES (query: step + method titles) ROUGE-1 38.3 # 3
Document Summarization HowSumm-Step LexRank (query: step + method titles) ROUGE-1 38.2 # 4
Document Summarization HowSumm-Step GreedyRel (query: step + method titles) ROUGE-1 30.3 # 7
Document Summarization HowSumm-Step BM25-HierSumm (query: step title) ROUGE-1 22.3 # 10

Methods


No methods listed for this paper. Add relevant methods here