Evaluating Zero-Shot Classification for Fake News Detection in the Libyan Dialect Using XLM-RoBERTa: An Exploratory Study
DOI:
https://doi.org/10.65405/sjh.2.3.266Keywords:
Fake News Detection; Libyan Dialect; Zero-Shot Classification; XLM-RoBERTa; Natural Language Processing; Model Evaluation.Abstract
This exploratory study evaluates the outputs of XLM-RoBERTa-Large-XNLI for zero-shot classification of fake and genuine news in the Libyan dialect. Six short texts with manually assigned reference labels were examined: two in Modern Standard Arabic and four in Libyan Arabic. The experiment was reported as running with Transformers in Google Colab, without task-specific fine-tuning. The original results table shows agreement between model predictions and reference labels in five cases and disagreement in one. Reported model scores include 50.5% for one dialectal text and 65.0% for the misclassified case. These scores are classifier confidence values, not accuracy estimates or calibrated probabilities of truth. The sample is too small to establish a statistically reliable dialectal performance gap. Future work should verify source-level ground truth, compile a larger balanced Libyan dataset, and compare zero-shot and fine-tuned baselines.
References
Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (pp. 7088–7105). Association for Computational Linguistics. https://aclanthology.org/2021.acl-long.551/
AlZawam, H. A., & Ahmad, T. M. A. (2026). Strengthening support vector machine capability through Harris Hawks optimization for Arabic text categorization. Al-Farooq Journal of Sciences, 2(4), 675–685. https://doi.org/10.65405/nsmnf959
Antoun, W., Baly, F., & Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (pp. 9–15). European Language Resources Association. https://aclanthology.org/2020.osact-1.2/
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 8440–8451). Association for Computational Linguistics. https://aclanthology.org/2020.acl-main.747/
Garash, S. S. (2025). A comparative study of Arabic text classification using k-NN, SVM, and Naive Bayes. Comprehensive Journal of Science, 10(37), 867–879. https://doi.org/10.65405/.v10i37.326
Haouari, F., Hasanain, M., Suwaileh, R., & Elsayed, T. (2021). ArCOV19-Rumors: Arabic COVID-19 Twitter dataset for misinformation detection. In Proceedings of the Sixth Arabic Natural Language Processing Workshop (pp. 72–83). Association for Computational Linguistics. https://aclanthology.org/2021.wanlp-1.8/
Himdi, H., Weir, G., Assiri, F., & Al-Barhamtoshy, H. (2022). Arabic fake news detection based on textual analysis. Arabian Journal for Science and Engineering, 47(8), 10453–10469. https://doi.org/10.1007/s13369-021-06449-y
Khalil, A., Jarrah, M., Aldwairi, M., & Jaradat, M. (2022). AFND: Arabic fake news dataset for the detection and classification of articles credibility. Data in Brief, 42, 108141. https://doi.org/10.1016/j.dib.2022.108141
Najadat, H., Tawalbeh, M., & Awawdeh, R. (2022). Fake news detection for Arabic headlines-articles news data using deep learning. International Journal of Electrical and Computer Engineering, 12(4), 3951–3959. https://doi.org/10.11591/ijece.v12i4.pp3951-3959
Shu, K., Sliva, A., Wang, S., Tang, J., & Liu, H. (2017). Fake news detection on social media: A data mining perspective. ACM SIGKDD Explorations Newsletter, 19(1), 22–36. https://doi.org/10.1145/3137597.3137600
Yin, W., Hay, J., & Roth, D. (2019). Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 3914–3923). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1404
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Shihab Journal of Humanities

This work is licensed under a Creative Commons Attribution 4.0 International License.
License
Copyright (c) 2026 Shihab Journal of Humanities

This work is licensed under a Creative Commons Attribution 4.0 International License.










