HowSumm: A Multi-Document Summarization Dataset Derived from WikiHow Articles

7 Oct 2021  ·  Odellia Boni, Guy Feigenblat, Guy Lev, Michal Shmueli-Scheuer, Benjamin Sznajder, David Konopnicki ·

We present HowSumm, a novel large-scale dataset for the task of query-focused multi-document summarization (qMDS), which targets the use-case of generating actionable instructions from a set of sources. This use-case is different from the use-cases covered in existing multi-document summarization (MDS) datasets and is applicable to educational and industrial scenarios. We employed automatic methods, and leveraged statistics from existing human-crafted qMDS datasets, to create HowSumm from wikiHow website articles and the sources they cite. We describe the creation of the dataset and discuss the unique features that distinguish it from other summarization corpora. Automatic and human evaluations of both extractive and abstractive summarization models on the dataset reveal that there is room for improvement.

PDF Abstract

Datasets

Introduced in the Paper:

HowSumm

Used in the Paper:

WikiHow Multi-News WikiSum WCEP Multi-XScience
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Document Summarization HowSumm-Method LexRank (query: method + article + steps titles) ROUGE-1 53.5 # 1
Compare
Document Summarization HowSumm-Method GreedyRel (query: method + article + steps titles) ROUGE-1 48.6 # 3
Compare
Document Summarization HowSumm-Method CES (query: method + article titles) ROUGE-1 48.3 # 5
Compare
Document Summarization HowSumm-Method LexRank (query: method + article titles) ROUGE-1 47.1 # 7
Compare
Document Summarization HowSumm-Method GreedyRel (query: method + article titles) ROUGE-1 42.3 # 9
Compare
Document Summarization HowSumm-Method CES (query: method title) ROUGE-1 48.4 # 4
Compare
Document Summarization HowSumm-Method LexRank (query: method title) ROUGE-1 47.7 # 6
Compare
Document Summarization HowSumm-Method GreedyRel (query: method title) ROUGE-1 43.4 # 8
Compare
Document Summarization HowSumm-Method CES (query: method + article + steps titles) ROUGE-1 52.2 # 2
Compare
Document Summarization HowSumm-Step CES (query: step title) ROUGE-1 39.3 # 2
Compare
Document Summarization HowSumm-Step LexRank (query: step title) ROUGE-1 39.6 # 1
Compare
Document Summarization HowSumm-Step GreedyRel (query: step title) ROUGE-1 30.1 # 8
Compare
Document Summarization HowSumm-Step BM25-HierSumm (query: step + method + article titles) ROUGE-1 21.9 # 11
Compare
Document Summarization HowSumm-Step CES (query: step + method + article titles) ROUGE-1 37.0 # 5
Compare
Document Summarization HowSumm-Step LexRank (query: step + method + article titles) ROUGE-1 36.3 # 6
Compare
Document Summarization HowSumm-Step BM25-HierSumm (query: step + method titles) ROUGE-1 23.0 # 9
Compare
Document Summarization HowSumm-Step CES (query: step + method titles) ROUGE-1 38.3 # 3
Compare
Document Summarization HowSumm-Step LexRank (query: step + method titles) ROUGE-1 38.2 # 4
Compare
Document Summarization HowSumm-Step GreedyRel (query: step + method titles) ROUGE-1 30.3 # 7
Compare
Document Summarization HowSumm-Step BM25-HierSumm (query: step title) ROUGE-1 22.3 # 10
Compare

Methods
Add Remove

No methods listed for this paper. Add relevant methods here
Contact us on: hello@paperswithcode.com . Papers With Code is a free resource with all data licensed under CC-BY-SA.
Terms Data policy Cookies policy