Overview of the 2022 Validity and Novelty Prediction Shared Task
This paper provides an overview of the Argument Validity and Novelty Prediction Shared Task that was organized as part of the 9th Workshop on Argument Mining (ArgMining 2022). The task focused on the prediction of the validity and novelty of a conclusion given a textual premise. Validity is defined as the degree to which the conclusion is justified with respect to the given premise. Novelty defines the degree to which the conclusion contains content that is new in relation to the premise. Six groups participated in the task, submitting overall 13 system runs for the subtask of binary classification and 2 system runs for the subtask of relative classification. The results reveal that the task is challenging, with best results obtained for Validity prediction in the range of 75% F1 score, for Novelty prediction of 70% F1 score and for correctly predicting both Validity and Novelty of 45% F1 score. In this paper we summarize the task definition and dataset. We give an overview of the results obtained by the participating systems, as well as insights to be gained from the diverse contributions.
PDF Abstract
Datasets
ConceptNet
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Benchmark |
|---|---|---|---|---|---|---|
| ValNov | ValNov Subtask A | ACCEPT-1 | JOINT-F1 | 43.13 | # 3 | |
| VAL-F1 | 59.20 | # 7 | ||||
| NOV-F1 | 70.00 | # 1 | ||||
| ValNov | ValNov Subtask A | NLP@UIT | JOINT-F1 | 25.89 | # 6 | |
| VAL-F1 | 61.72 | # 5 | ||||
| NOV-F1 | 43.36 | # 6 | ||||
| ValNov | ValNov Subtask A | CSS | JOINT-F1 | 42.40 | # 4 | |
| VAL-F1 | 70.76 | # 2 | ||||
| NOV-F1 | 59.86 | # 4 | ||||
| ValNov | ValNov Subtask A | Harshad | JOINT-F1 | 17.35 | # 8 | |
| VAL-F1 | 56.31 | # 8 | ||||
| NOV-F1 | 39.00 | # 7 | ||||
| ValNov | ValNov Subtask A | Baseline | JOINT-F1 | 23.90 | # 7 | |
| VAL-F1 | 59.96 | # 6 | ||||
| NOV-F1 | 36.12 | # 8 | ||||
| ValNov | ValNov Subtask A | CLTeamL-3 | JOINT-F1 | 45.16 | # 1 | |
| VAL-F1 | 74.64 | # 1 | ||||
| NOV-F1 | 61.75 | # 3 | ||||
| ValNov | ValNov Subtask A | System Average | JOINT-F1 | 35.94 | # 5 | |
| VAL-F1 | 62.74 | # 4 | ||||
| NOV-F1 | 52.97 | # 5 | ||||
| ValNov | ValNov Subtask B | AXiS@EdUni | JOINT-F1 | 29.16 | # 2 | |
| NOV-F1 | 25.86 | # 2 | ||||
| VAL-F1 | 32.47 | # 2 | ||||
| ValNov | ValNov Subtask B | NLP@UIT | JOINT-F1 | 41.50 | # 1 | |
| NOV-F1 | 38.39 | # 1 | ||||
| VAL-F1 | 44.60 | # 1 | ||||
| ValNov | ValNov Subtask B | Baseline | JOINT-F1 | 21.46 | # 3 | |
| NOV-F1 | 23.09 | # 3 | ||||
| VAL-F1 | 19.82 | # 3 |
