Challenge or Empower: Revisiting Argumentation Quality in a News Editorial Corpus

News editorials are said to shape public opinion, which makes them a powerful tool and an important source of political argumentation. However, rarely do editorials change anyone{'}s stance on an issue completely, nor do they tend to argue explicitly (but rather follow a subtle rhetorical strategy). So, what does argumentation quality mean for editorials then? We develop the notion that an effective editorial challenges readers with opposing stance, and at the same time empowers the arguing skills of readers that share the editorial{'}s stance {---} or even challenges both sides. To study argumentation quality based on this notion, we introduce a new corpus with 1000 editorials from the New York Times, annotated for their perceived effect along with the annotators{'} political orientations. Analyzing the corpus, we find that annotators with different orientation disagree on the effect significantly. While only 1{\%} of all editorials changed anyone{'}s stance, more than 5{\%} meet our notion. We conclude that our corpus serves as a suitable resource for studying the argumentation quality of news editorials.