Dataset Creation for Ranking Constructive News Comments
Ranking comments on an online news service is a practically important task for the service provider, and thus there have been many studies on this task. However, most of them considered users{'} positive feedback, such as {``}Like{''}-button clicks, as a quality measure. In this paper, we address directly evaluating the quality of comments on the basis of {``}constructiveness,{''} separately from user feedback. To this end, we create a new dataset including 100K+ Japanese comments with constructiveness scores (C-scores). Our experiments clarify that C-scores are not always related to users{'} positive feedback, and the performance of pairwise ranking models tends to be enhanced by the variation of comments rather than articles.
PDF Abstract