OVERVIEW OF MULTIMODAL DATA AND ITS APPLICATION TO FAKE-NEWS DETECTION

(Received: 29-Feb.-2024, Revised: 26-Apr.-2024 , Accepted: 14-May-2024)

Authors Nataliya Boyko,

Keywords #Technologies #Information environment #Neural networks #Testing approaches #Disinformation sources

Abstract In the context of the growing popularity of social media over the past ten years, an urgent problem of fake news spreading has arisen, which underscores the research’s relevance. The aim of this article is to assess the efficacy of multimodal approaches in detecting fake news, a pressing issue given the substantial impact misinformation can have on health, politics and economics. To achieve this goal, a multimodal approach was chosen that combines deep-learning frameworks and pre-trained models. This approach provides a comprehensive analysis of textual, visual and audio information, allowing for more accurate identification of disinformation sources. The use of various knowledge-transfer methods made it possible to process information efficiently, improving the quality of classification. The study conducted a thorough analysis of various data-collection strategies, as well as a comparative analysis of available multimodal approaches to fake-news detection and the datasets used. The results of this study included a detailed analysis of current research work in the field of fake-news detection and the development of a multimodal approach to this problem. Textual, visual and audio information was processed using pre-trained models and deep learning, achieving high accuracy in fake news detection. The results of the study indicated that the multimodal approach allows for more accurate identification of sources of disinformation and increases the efficiency of fake-news classification compared to other methods. A comparative analysis of various data collection strategies and datasets was also conducted, confirming the high efficiency of the approach under various conditions.

References

[1] G. Di Domenico and M. Visentin, "Fake News or True Lies? Reflections about Problematic Contents in Marketing," International Journal of Market Research, vol. 62, no. 4, pp. 409-417, 2020.

[2] J. C. Culpepper, "Merriam-Webster Online: The Language Center," Electronic Resources Review, vol. 4, no. 1/2, pp. 9-11, 2000.

[3] O. Ajao, D. Bhowmik and S. Zargari, "Sentiment Aware Fake News Detection on Online Social Networks," Proc. of the ICASSP 2019 – 2019 IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp. 2507-2511, Brighton, UK, 2019.

[4] H. Allcott and M. Gentzkow, "Social Media and Fake News in the 2016 Election," Journal of Economic Perspectives, vol. 31, no. 2, pp. 211-236, 2017.

[5] A. K. Dubey and M. Saraswat, "Fake News Detection through ML and Deep Learning Approaches for Better Accuracy," Proc. of Advances in Computational Intelligence and Communication Technology (CICT 2021), Part of the Book Series: Lecture Notes in Networks and Systems, vol. 399, pp. 13-21, 2022.

[6] C. M. Greene and G. Murphy, "Quantifying the Effects of Fake News on Behavior: Evidence from a Study of COVID-19 Misinformation," J. of Experimental Psychology: Applied, vol. 27, no. 4, pp. 773-784, 2021.

[7] M. S. Islam et al., "COVID-19 Vaccine Rumors and Conspiracy Theories: The Need for Cognitive Inoculation against Misinformation to Improve Vaccine Adherence," PLoS ONE, vol. 16, no. 5, Article no. e0251605, 2021.

[8] E. Brown, "Online Fake News is Costing us $78 Billion Globally Each Year," ZD NET, [Online], Available: https://www.zdnet.com/article/online-fake-news-costing-us-78-billion-globally-each-year/, 2019.

[9] O. P. Prosyanyk and S. G. Holovnia, "Methods of Detecting Fake News in Social Networks," Proc. of the Int. Conf. on Scientific and Practical Developments (The European Development Trends in Journalism, PR, Media and Communication), pp. 81-85, 2021.

[10] D. O. Tatarchuk, "Fact-checking Tools for Detecting Fake Information in Social Media," Proc. of the Int. Conf. on Scientific and Practical Developments (The European Development Trends in Journalism, PR, Media and Communication), pp. 84-86, 2020.

[11] I. Ivanova and O. Lysytskaia, "Postmodernism As a Manipulative Technology in Modern Ukrainian Advertising: The Artistic Dominant Characteristic," Int. J. of Philology, vol. 11, no. 1, pp. 108-113, 2020.

[12] Y. F. Shtefaniuk, I. R. Opirskyy and O. I. Harasymchuk, "Analysis of Application of Existing Fake News Recognition Techniques to Counter Information Propaganda," Ukrainian Scientific J. of Information Security, vol. 26, no. 3, pp. 139-144, 2020.

[13] V. Bazylevych and M. Prybytko, "Fake News Detection System Based on Data Science," Technical Sciences and Technologies, vol. 4, no. 22, pp. 91-95, 2020.

[14] A. Thota, P. Tilak, S. Ahluwalia and N. Lohia, "Fake News Detection: A Deep Learning Approach," SMU Data Science Review, vol. 1, no. 3, pp. 1-10, 2018.

[15] L. Donatelli, N. Krishnaswamy, K. Lai and J. Pustejovsky, Proc. of the 1st Workshop on Multimodal Semantic Representations (MMSR), Association for Computat. Linguist., Groningen, Netherlands, 2021.

[16] V. K. Singh, I. Ghosh and D. Sonagara, "Detecting Fake News Stories via Multimodal Analysis," Journal of the Association for Information Science and Technology, vol. 72, no. 1, pp. 3-17, 2021.

[17] A. Giachanou, G. Zhang and P. Rosso, "Multimodal Multi-image Fake News Detection," Proc. of the 2020 IEEE 7th Int. Conf. on Data Science and Advanced Analytics (DSAA), pp. 647-654, Sydney, Australia, 2020.

[18] Y. Khimich, "Formation of Information Culture of Higher Education Students in the Digital Age," Library Science-Record Studies-Informology, vol. 1, no. 1, pp. 86-95, DOI: 10.32461/2409-9805.1.2023.276773, 2023.

[19] H. Ahmed, I. Traore and S. Saad, "Detection of Online Fake News Using N-gram Analysis and Machine Learning Techniques," Proc. of the 1st Int. Conf. on Intelligent, Secure and Dependable Systems in Distributed and Cloud Environments (ISDDC 2017), Part of the Book Series: Lecture Notes in Computer Science, vol. 10618, pp. 127-138, 2017.

[20] D. Kumar Sharma and S. Sharma, "Comment Filtering-based Explainable Fake News Detection," Proc. of 2nd Int. Conf. on Computing, Communications and Cyber-security, Part of the Book Series: Lecture Notes in Networks and Systems, vol. 203, pp. 447-458, 2021.

[21] S. Garg and D. Kumar Sharma, "New Politifact: A Dataset for Counterfeit News," Proc. of the 2020 9th Int. Conf. System Modeling and Advancement in Research Trends (SMART), pp. 17-22, Moradabad, India, 2020.

[22] H. Karimi, P. Roy, S. Saba-Sadiya and J. Tang, "Multi-source Multi-class Fake News Detection," Proc. of the 27th IEEE Int. Conf. on Computational Linguistics, pp. 1546-1557, Santa Fe, New Mexico, USA, 2018.

[23] R. Oshikawa et al., "A Survey on Natural Language Processing for Fake News Detection," Proc. of the 12th Language Resources and Evaluation Conf., pp. 6086-6093, Marseille, France, 2020.

[24] T. Martyniuk, O. Voytsekhovska, M. Ochkurov and O. Voinalovych, "Properties of Unit Encoding of Information in the Context of Functional Control," ІТКІ, vol. 57, no. 2, pp. 43-49, 2023.

[25] V. Varenko, "Electronic Communications in Information and Analytical Activities," Library Science- Record Studies- Informology, vol. 1, no. 1, pp. 53-58, DOI: 10.32461/2409-9805.1.2023.276765, 2023.

[26] S. Singhal et al., "SpotFake: A Multi-modal Framework for Fake News Detection," Proc. of the 2019 IEEE 5th Int. Conf. on Multimedia Big Data (BigMM), pp. 39-47, Singapore, 2019.

[27] Y. Yang, L. Zheng, J. Zhang, Q. Cui, Z. Li and P.S. Yu, "TI-CNN: Convolutional Neural Networks for Fake News Detection," arXiv: 1806.00749v3, DOI: 10.48550/arXiv.1806.00749, 2018.

[28] J. Ma, W. Gao and K. F. Wong, "Rumor Detection on Twitter with Tree-structured Recursive Neural Networks," Proc. of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 1: Long Papers, pp. 1980-1989, Melbourne, Australia, 2018.

[29] Q. You, L. Cao, H. Jin and J. Luo, "Robust Visual-textual Sentiment Analysis: When Attention Meets Tree-structured Recursive Neural Networks," Proc. of the 24th ACM Int. Conf. on Multimedia (MM’16), pp. 1008-1017, DOI: 10.1145/2964284.2964288, 2016.

[30] P. Kumar Verma, P. Agrawal, V. Madaan and R. Prodan, "MCred: Multi-modal Message Credibility for Fake News Detection Using BERT and CNN," Journal of Ambient Intelligence and Humanized Computing, vol. 14, pp. 10617-10629, 2022.

[31] K. Sharifani, M. Amini, Y. Akbari and J. A. Godarzi, "Operating Machine Learning across Natural Language Processing Techniques for Improvement of Fabricated News Model," Int. Journal of Science and Information System Research, vol. 12, no. 9, pp. 20-44, 2022.

[32] Y. Wang, S. Qian, J. Hu, Q. Fang and C. Xu, "Fake News Detection via Knowledge-driven Multimodal Graph Convolutional Networks," Proc. of the 2020 Int. Conf. on Multimedia Retrieval (ICMR’20), pp. 540-547, DOI: 10.1145/3372278.3390713, 2020.

[33] C. Song, N. Ning, Y. Zhang and B. Wu, "A Multimodal Fake News Detection Model Based on Crossmodal Attention Residual and Multichannel Convolutional Neural Networks," Information Processing & Management, vol. 58, no. 1, Article no. 102437, 2021.

[34] I. Goodfellow et al., "Generative Adversarial Networks," Communications of the ACM, vol. 63, no. 11, pp. 139-144, 2020.

[35] J. Du et al., "Cross-lingual COVID-19 Fake News Detection," Proc. of the 2021 Int. Conf. on Data Mining Workshops (ICDMW), pp. 859-862, DOI: 10.1109/ICDMW53433.2021.00110, 2021.

[36] J. Wang, M. Gao, Y. Huang, K. Shu and H. Yi, "FinD: Fine-grained Discrepancy-based Fake News Detection Enhanced by Event Abstract Generation," Computer Speech & Language, vol. 78, Article no. 101461, DOI: 10.1016/j.csl.2022.101461, 2023.

[37] S. Xiong, G. Zhang, V. Batra, L. Xi, L. Shi and L. Liu, "TRIMOON: Two-round Inconsistency-based Multi-modal Fusion Network for Fake News Detection," Information Fusion, vol. 93, pp. 150-158, 2023.

[38] J. Xue, H. Zhou, H. Song, B. Wu and L. Shi, "Cross-modal Information Fusion for Voice Spoofing Detection," Speech Communication, vol. 147, pp. 41-50, DOI: 10.1016/j.specom.2023.01.001, 2023.

[39] J. Hua, X. Cui, X. Li, K. Tang and P. Zhu, "Multimodal Fake News Detection through Data Augmentation-based Contrastive Learning," Applied Soft Computing, vol. 136, Article no. 110125, DOI: 10.1016/j.asoc.2023.110125, 2023.

[40] R. N. Kvyetnyy, O. N. Romanyuk, E. O. Titarchuk, K. Gromaszek and N. Mussabekov, "Usage of the Hybrid Encryption in a Cloud Instant Messages Exchange System," Proc. of SPIE – The Int. Society for Optical Engineering, vol. 10031, Article no. 100314S, DOI: 10.1117/12.2249190, 2016.

[41] E. Ginters and E. Dimitrovs, "Latent Impacts on Digital Technologies Sustainability Assessment and Development," Advances in Intelligent Systems and Computing, vol. 1365, pp. 3-13, DOI: 10.1007/978-3-030-72657-7_1, 2021.

[42] E. Ginters, "New Trends towards Digital Technology Sustainability Assessment," Proc. of the World Conf. on Smart Trends in Systems, Security and Sustainability, vol. WS4 2020, pp. 184-189, DOI: 10.1109/WorldS450073.2020.9210408, 2020.

[43] M. Vasylkivskyi, O. Horodetska, B. Klymchuk and V. Hovorun, "Strategies of Technological Development of Hardware of Infocommunication Radio Networks," ІТКІ, vol. 56, no. 1, pp. 83-91, 2023.

[44] J. Xue, Y. Wang, Y. Tian, Y. Li, L. Shi and L. Wei, "Detecting Fake News by Exploring the Consistency of Multimodal Data," Information Processing & Management, vol. 58, no. 5, Article no. 102610, 2021.