INTERPRETING ARABIC TRANSFORMER MODELS: A STUDY ON XAI INTERPRETABILITY FOR QUR’ANIC SEMANTIC-SEARCH MODELS

(Received: 10-Jan.-2024, Revised: 24-Mar.-2024, 16-May-2024 and 8-Jul.-2024 , Accepted: 13-Jul.-2024)

Authors STUDY ON XAI INTERPRETABILITY FOR QUR’ANIC Ahmad M. Mustafa, Saja Nakhleh, Rama Irsheidat, Raneem Alruosan,

Keywords #Explainable machine learning #Semantic search #Arabic NLP #Transformers #SHAP #LIME

Abstract Transformers have shown their effectiveness in various machine-learning tasks. However, their “black box” nature often obscures their decision-making processes, particularly in Arabic, posing a barrier to their broader adoption and trust. This study delves into the interpretability of three Arabic transformer models that have been fine-tuned for semantic-search tasks. Through a focused case study, we employ these models for retrieving information from the Holy Qur’an, leveraging Explainable AI (XAI) techniques—namely, LIME and SHAP—to shed light on the decision-making processes of these models. The paper underscores the unique challenges posed by the Qur’anic text and demonstrates how XAI can significantly boost the transparency and interpretability of semantic-search systems for such complex text. Our findings reveal that applying XAI techniques to Arabic transformer models for Qur’anic content not only demystifies the models’ internal mechanics, but also makes the insights derived from them more accessible to a broader audience. This contribution is twofold: It enriches the field of XAI within the context of Arabic semantic search and illustrates the utility of these techniques in deepening our understanding of intricate religious documents. By providing this nuanced approach to the interpretability of Arabic transformer models in the domain of semantic search, our study underscores the potential of XAI to bridge the gap between advanced machine-learning technologies and the nuanced needs of users seeking to explore complex texts like the Holy Qur’an.

References

[1] A. Ali, T. Schnake, O. Eberle, G. Montavon, K.-R. Müller and L. Wolf, "XAI for Transformers: Better Explanations through Conservative Propagation," Proc. of the 39th Int. Conf. on Machine Learning, ser. Proc. of Machine Learning Research, vol. 162, pp. 435–451, PMLR, [Online], Available: https://proceedings.mlr.press/v162/ali22a.html, 17–23 Jul 2022.

[2] W. Saeed and C. Omlin, "Explainable AI (XAI): A Systematic Meta-survey of Current Challenges and Future Opportunities," Knowledge-based Systems, vol. 263, p. 110273, 2023.

[3] H. U. Khan, S. M. Saqlain, M. Shoaib and M. Sher, "Ontology Based Semantic Search in Holy Quran," International Journal of Future Computer and Communication, vol. 2, no. 6, p. 570, 2013.

[4] I. Al-Huri et al., "Arabic Language: Historic and Sociolinguistic Characteristics," English Literature and Language Review, vol. 1, no. 4, pp. 28–36, 2015.

[5] M. Mustafa, H. AbdAlla and H. Suleman, "Current Approaches in Arabic IR: A Survey," Proc. of the Int. Conf. on Asian Digital Libraries, Digital Libraries: Universal and Ubiquitous Access to Information, Part of the Book Series: Lecture Notes in Comp. Science, vol. 5362, pp. 406–407, 2008.

[6] E. H. Mohamed and E. M. Shokry, "QSST: A Quranic Semantic Search Tool Based on Word Embedding," J. of King Saud Uni.-Computer and Inform. Sciences, vol. 34, no. 3, pp. 934–945, 2022.

[7] R. Malhas and T. Elsayed, "Arabic Machine Reading Comprehension on the Holy Qur’an Using Clarabert," Information Processing and Management, vol. 59, no. 6, p. 103068, DOI:10.1016/j.ipm.2022.103068, 2022.

[8] A. Vaswani et al., "Attention Is All You Need," arXiv: 1706.03762, DOI: 10.48550/arXiv.1706.03762, 2023.

[9] A. Safaya, M. Abdullatif and D. Yuret, "KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media," Proc. of the 14th Workshop on Semantic Evaluation, Barcelona (online): Int. Committee for Computational Linguistics, pp. 2054–2059, Dec. 2020.

[10] N. Reimers and I. Gurevych, "Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation," Proc. of the 2020 Conf. on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, arXiv: 2004.09813, 2020.

[11] E. Kokalj, B. Škrlj, N. Lavrač, S. Pollak and M. Robnik-Šikonja, "BERT Meets Shapley: Extending SHAP Explanations to Transformer-based Classifiers," Proc. of the EACL Hackashop on News Media Content Analysis and Automated Report Generation, Association for Computational Linguistics, pp. 16–21, [Online]. Available: https://aclanthology.org/2021.hackashop-1.3, Apr. 2021.

[12] S. M. Lundberg and S.-I. Lee, "A Unified Approach to Interpreting Model Predictions," Proc. of the 31st Int. Conf. on Neural Information Processing Systems (NIPS’17), pp. 4768–4777, Red Hook, USA, 2017.

[13] M. T. Ribeiro, S. Singh and C. Guestrin, "”Why Should I Trust You?”: Explaining the Predictions of Any Classifier," Proc. of the 22nd ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD ’16), pp. 1135–1144, New York, NY, USA, DOI: 10.1145/2939672.2939778, 2016.

[14] M. H. Bashir et al., "Arabic Natural Language Processing for Qur’anic Research: A Systematic Review," Artificial Intelligence Review, vol. 56, no. 7, pp. 6801–6854, 2022.

[15] A. Farghaly and K. Shaalan, "Arabic Natural Language Processing: Challenges and Solutions," ACM Trans. on Asian Language Inform. Processing, vol. 8, no. 4, DOI: 10.1145/1644879.1644881, 2009.

[16] K. Dukes and N. Habash, "Morphological Annotation of Quranic Arabic," Proc. of the 7th Int. Conf. on Language Resources and Evaluation (LREC’10), Valletta, Malta, ELRA, May 2010.

[17] S. Altammami and E. Atwell, "Challenging the Transformer-based Models with a Classical Arabic Dataset: Quran and Hadith," Proceedings of the 13th Language Resources and Evaluation Conf., pp. 1462–1471, European Language Resources Association, Marseille, France, Jun. 2022.

[18] D. Vale, A. El-Sharif and M. Ali, "Explainable Artificial Intelligence (XAI) Post-hoc Explainability Methods: Risks and Limitations in Non-discrimination Law," AI and Ethics, vol. 2, no. 4, pp. 815–826, 2022.

[19] E. M. Kenny, E. D. Delaney, D. Greene and M. T. Keane, "Post-hoc Explanation Options for XAI in Deep Learning: The Insight Centre for Data Analytics Perspective," Proc. of Int. Conf. on Pattern Recognition, ICPR Int. Workshops and Challenges, Part of the Book Series: Lecture Notes in Computer Science, vol. 12663, pp. 20–34, 2021.

[20] A. Sarkar, D. Vijaykeerthy, A. Sarkar and V. N. Balasubramanian, "A Framework for Learning Ante-hoc Explainable Models via Concepts," Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 10 286–10 295, June 2022.

[21] A. Alsaleh, E. Atwell and A. Altahhan, "Quranic Verses Semantic Relatedness Using AraBERT," Proc. of the 6th Arabic Natural Language Processing Workshop, pp. 185–190, Kyiv, Ukraine, [Online]. Available: https://aclanthology.org/2021.wanlp-1.19, Apr. 2021.

[22] S. Saeed, S. Haider and Q. Rajput, "On Finding Similar Verses from the Holy Quran Using Word Embeddings," Proc. of the 2020 IEEE Int. Conf. on Emerging Trends in Smart Technologies (ICETST), pp. 1–6, Karachi, Pakistan, 2020.

[23] S. M. Lundberg et al., "Explainable Machine-learning Predictions for the Prevention of Hypoxaemia During Surgery," Nature Biomedical Engineering, vol. 2, no. 10, p. 749, 2018.

[24] C. Leiter, P. Lertvittayakumjorn, M. Fomicheva, W. Zhao, Y. Gao and S. Eger, "Towards Explainable Evaluation Metrics for Machine Translation," Journal of Machine Learning Research, vol. 25, pp. 1-49, 2023.

[25] J. El Zini, M. Mansour, B. Mousi and M. Awad, "On the Evaluation of the Plausibility and Faithfulness of Sentiment Analysis Explanations," Proc. of the IFIP Int. Conf. on Artificial Intelligence Applications and Innovations, Part of the Book Series: IFIP Advances in Information and Communication Technology, vol. 647, pp. 338–349, 2022.

[26] N. Habash, Introduction to Arabic Natural Language Processing, 1st Edn., ser. Synthesis Lectures on Human Language Technologies, Morgan and Claypool Publishers, vol. 3, pp. 1-124, 2010.

[27] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger and Y. Artzi, "BERTscore: Evaluating Text Generation with BERT," Towards Data Science, arXiv: 1904.09675, 2020.

[28] Y. Qiao, C. Xiong, Z. Liu and Z. Liu, "Understanding the Behaviors of BERT in Ranking," arXiv preprint, arXiv: 1904.07531, 2019.

[29] S. Yagi, A. Elnagar and S. Fareh, "A Benchmark for Evaluating Arabic Word Embedding Models," Natural Language Engineering, vol. 29, no. 4, p. 978–1003, 2023.

[30] B. Dahy, M. Farouk and K. Fathy, "Arabic Sentences Semantic Similarity Based on Word Embedding," Proc. of the 2022 20th Int. Conf. on Language Engineering (ESOLEC), vol. 20, pp. 35–40, Valencia, Spain, 2022.

[31] G. Salton, "Introduction to Modern Information Retrieval," ISBN 0-07-054484-0, McGraw-Hill, 1983.