Multimodal Fake News Detection in Social Media using Transformer Models with Text Image

Abstract

Fake news on social media increasingly combines misleading textual narratives with manipulated or unrelated images, making detection more challenging for unimodal approaches. This study proposes a transformer-based multimodal fake news detection framework that integrates BERT for textual feature extraction and Vision Transformer (ViT) for visual representation through feature-level fusion. The proposed framework aims to capture complementary semantic and contextual information from both modalities to improve fake news classification performance. We evaluate the model using standard classification metrics and compare it with unimodal baseline models under identical experimental settings. Experimental results show that the proposed multimodal framework achieves 91.3% accuracy, 90.5% precision, 92.4% recall, and 91.4% F1-score, outperforming text-only and image-only models across all evaluation metrics. The findings demonstrate that multimodal fusion effectively captures cross-modal inconsistencies and improves classification robustness in social media environments. Overall, this study confirms that transformer-based multimodal learning provides an effective and reliable solution for fake news detection and contributes to the development of intelligent misinformation monitoring systems.