Comparative evaluation of attention-based early fusion techniques for fine-grained multimodal fake news detection

Abstract

Abstract

The rapid proliferation of misinformation across online platforms has intensified the need for reliable automated fake news detection systems, particularly those capable of processing multimodal content that combines textual and visual information. This study investigates the effectiveness of attention-based fusion mechanisms, namely self-attention (SA), multi-head attention (MHA), and co-attention (CoAtt), for enhancing multimodal fake news detection through the integration of textual features extracted using bidirectional encoder representations from transformers (BERT) and visual features derived from residual network (ResNet) architectures. Two model configurations, BERT+ResNet18 and BERT+ResNet50, were evaluated using a fine-grained multimodal dataset comprising six news categories: true, satire, misleading, manipulated, false connection, and imposter content. Feature fusion was performed at the representation level, and attention mechanisms were employed to improve cross-modal feature interactions and representation learning. Model performance was evaluated using accuracy, precision, recall, and F1-score metrics. The experimental results demonstrate that MHA consistently outperforms the other attention-based fusion methods and remains competitive with conventional fusion approaches across both model configurations, achieving accuracies of 73% and 74% for the BERT+ResNet18 and BERT+ResNet50 models, respectively. Furthermore, per-class analysis reveals that MHA provides a more balanced classification performance across both majority and minority classes, whereas SA and CoAtt exhibit reduced effectiveness when handling underrepresented categories. Overall, the findings suggest that attention-based fusion strategies, particularly MHA, provide a robust and effective framework for fine-grained multimodal fake news detection.