On the Benefit of Combining Neural, Statistical and External Features for Fake News Identification

Identifying the veracity of a news article is an interesting problem while automating this process can be a challenging task. Detection of a news article as fake is still an open question as it is contingent on many factors which the current state-of-the-art models fail to incorporate. In this paper, we explore a subtask to fake news identification, and that is stance detection. Given a news article, the task is to determine the relevance of the body and its claim. We present a novel idea that combines the neural, statistical and external features to provide an efficient solution to this problem. We compute the neural embedding from the deep recurrent model, statistical features from the weighted n-gram bag-of-words model and handcrafted external features with the help of feature engineering heuristics. Finally, using deep neural layer all the features are combined, thereby classifying the headline-body news pair as agree, disagree, discuss, or unrelated. We compare our proposed technique with the current state-of-the-art models on the fake news challenge dataset. Through extensive experiments, we find that the proposed model outperforms all the state-of-the-art techniques including the submissions to the fake news challenge.

PDF Abstract

Datasets


Results from the Paper


Task Dataset Model Metric Name Metric Value Global Rank Uses Extra
Training Data
Result Benchmark
Fake News Detection FNC-1 Bhatt et al. Weighted Accuracy 83.08 # 3
Per-class Accuracy (Agree) 43.82 # 6
Per-class Accuracy (Disagree) 6.31 # 6
Per-class Accuracy (Discuss) 85.68 # 3
Per-class Accuracy (Unrelated) 98.04 # 2
Fake News Detection FNC-1 Baseline based on skip-thought embeddings (Bhatt et al., 2017) Weighted Accuracy 76.18 # 8
Per-class Accuracy (Agree) 31.80 # 8
Per-class Accuracy (Disagree) 0.00 # 8
Per-class Accuracy (Discuss) 81.20 # 6
Per-class Accuracy (Unrelated) 91.18 # 7
Fake News Detection FNC-1 Baseline based on word2vec + hand-crafted features (Bhatt et al., 2017) Weighted Accuracy 72.78 # 9
Per-class Accuracy (Agree) 50.70 # 4
Per-class Accuracy (Disagree) 9.61 # 4
Per-class Accuracy (Discuss) 53.38 # 8
Per-class Accuracy (Unrelated) 96.05 # 5
Fake News Detection FNC-1 Neural baseline based on bi-directional LSTMs (Bhatt et al., 2017) Weighted Accuracy 63.11 # 10
Per-class Accuracy (Agree) 38.04 # 7
Per-class Accuracy (Disagree) 4.59 # 7
Per-class Accuracy (Discuss) 58.132 # 7
Per-class Accuracy (Unrelated) 78.27 # 8

Methods


No methods listed for this paper. Add relevant methods here