Evaluation of Language Models on Romanian XQuAD and RoITD datasets
DOI:
https://doi.org/10.15837/ijccc.2023.1.5111Keywords:
NLP, Question Answering, RoBert, RoGPT, DistilBert, TransformerAbstract
Natural language processing (NLP) has become a vital requirement in a wide range of applications, including machine translation, information retrieval, and text classification. The development and evaluation of NLP models for various languages have received significant attention in recent years, but there has been relatively little work done on comparing the performance of different language models on Romanian data. In particular, the introduction and evaluation of various Romanian language models with multilingual models have barely been comparatively studied. In this paper, we address this gap by evaluating eight NLP models on two Romanian datasets, XQuAD and RoITD. Our experiments and results show that bert-base-multilingual-cased and bertbase- multilingual-uncased, perform best on both XQuAD and RoITD tasks, while RoBERT-small model and DistilBERT models perform the worst. We also discuss the implications of our findings and outline directions for future work in this area.
References
Akbik, A., Chiticariu, L., Danilevsky, M., Li, Y., Vaithyanathan, S. & Zhu, H. Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling. Proceedings Of The 53rd Annual Meeting Of The Association For Computational Linguistics And The 7th International Joint Conference On Natural Language Processing (Volume 1: Long Papers). pp. 397-407 (2015,7), https://aclanthology.org/P15-1039
https://doi.org/10.3115/v1/P15-1039
Alyafeai, Z. & Ahmad, I. Arabic Compact Language Modelling for Resource Limited Devices. Proceedings Of The Sixth Arabic Natural Language Processing Workshop. pp. 53-59 (2021,4), https://aclanthology.org/2021.wanlp-1.6
Artetxe, M., Ruder, S. & Yogatama, D. On the Cross-lingual Transferability of Monolingual Representations. Annual Meeting Of The Association For Computational Linguistics. (2019)
https://doi.org/10.18653/v1/2020.acl-main.421
Asai, A., Eriguchi, A., Hashimoto, K. & Tsuruoka, Y. Multilingual Extractive Reading Comprehension by Runtime Machine Translation. ArXiv. abs/1809.03275 (2018)
Avram, A., Catrina, D., Cercel, D., Dascualu, M., Rebedea, T., Puaics, V. & Tufics, D. Distilling the Knowledge of Romanian BERTs Using Multiple Teachers. LREC. (2021)
Avram, A., Catrina, D., Cercel, D., Dascălu, M., Rebedea, T., Păiş, V. & Tufiş, D. Distilling the Knowledge of Romanian BERTs Using Multiple Teachers. (arXiv,2021)
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. & Amodei, D. Language Models Are Few-Shot Learners. Proceedings Of The 34th International Conference On Neural Information Processing Systems. (2020)
Chen, D., Fisch, A., Weston, J. & Bordes, A. Reading Wikipedia to Answer Open-Domain Questions. Proceedings Of The 55th Annual Meeting Of The Association For Computational Linguistics (Volume 1: Long Papers). pp. 1870-1879 (2017,7), https://aclanthology.org/P17- 1171
https://doi.org/10.18653/v1/P17-1171
Clark, K., Luong, M., Le, Q. & Manning, C. ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators. 8th International Conference On Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. (2020), https://openreview.net/forum?id=r1xMH1BtvB
Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H. & Stoyanov, V. XNLI: Evaluating Cross-lingual Sentence Representations. Proceedings Of The 2018 Conference On Empirical Methods In Natural Language Processing. pp. 2475-2485 (2018), https://aclanthology.org/D18-1269
https://doi.org/10.18653/v1/D18-1269
Cui, Y., Chen, Z., Wei, S., Wang, S., Liu, T. & Hu, G. Attention-over-Attention Neural Networks for Reading Comprehension. Proceedings Of The 55th Annual Meeting Of The Association For Computational Linguistics (Volume 1: Long Papers). pp. 593-602 (2017,7), https://aclanthology.org/P17-1055
https://doi.org/10.18653/v1/P17-1055
Devlin, J., Chang, M., Lee, K. & Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings Of The 2019 Conference Of The North American Chapter Of The Association For Computational Linguistics: Human Language Technologies, Volume 1 (Long And Short Papers). pp. 4171-4186 (2019)
Dhingra, B., Liu, H., Yang, Z., Cohen, W. & Salakhutdinov, R. Gated-Attention Readers for Text Comprehension. Proceedings Of The 55th Annual Meeting Of The Association For Computational Linguistics (Volume 1: Long Papers). pp. 1832-1846 (2017,7), https://aclanthology.org/P17-1168
https://doi.org/10.18653/v1/P17-1168
Dumitrescu, S., Avram, A. & Pyysalo, S. The birth of Romanian BERT. Findings Of The Association For Computational Linguistics: EMNLP 2020. pp. 4324-4328 (2020), https://aclanthology.org/2020.findings-emnlp.387
https://doi.org/10.18653/v1/2020.findings-emnlp.387
Ion R., Badea V.G., Cioroiu G., Mititelu V., Irimia E., Mitrofan M. & Tufis D. A Dialog Manager for Micro-Worlds In Studies in Informatics and Control, 29(4) . ISSN: 1220-1766 eISSN: 1841-429X pp. 411-420 (2020)
https://doi.org/10.24846/v29i4y202003
Hendrycks, D. & Gimpel, K. Gaussian Error Linear Units (GELUs). ArXiv: Learning. (2016)
Hinton, G., Vinyals, O. & Dean, J. Distilling the Knowledge in a Neural Network. ArXiv. abs/1503.02531 (2015)
Kadlec, R., Schmid, M., Bajgar, O. & Kleindienst, J. Text Understanding with the Attention Sum Reader Network. Proceedings Of The 54th Annual Meeting Of The Association For Computational Linguistics (Volume 1: Long Papers). pp. 908-918 (2016,8), https://aclanthology.org/P16-1086
https://doi.org/10.18653/v1/P16-1086
Kim, J., Jun, J. & Zhang, B. Bilinear Attention Networks. Proceedings Of The 32nd International Conference On Neural Information Processing Systems. pp. 1571-1581 (2018)
Kingma, D. & Ba, J. Adam: A Method for Stochastic Optimization. CoRR. abs/1412.6980 (2014)
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M., Dai, A., Uszkoreit, J., Le, Q. & Petrov, S. Natural Questions: A Benchmark for Question Answering Research. Transactions Of The Association For Computational Linguistics. 7 pp. 452-466 (2019), https://aclanthology.org/Q19-1026
https://doi.org/10.1162/tacl_a_00276
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P. & Soricut, R. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations.. ICLR. (2020)
Lee, S., Jang, H., Baik, Y., Park, S. & Shin, H. KR-BERT: A Small-Scale Korean-Specific Language Model. ArXiv: Computation And Language. (2020)
https://doi.org/10.5626/JOK.2020.47.7.682
Lewis, D., Yang, Y., Rose, T. & Li, F. RCV1: A New Benchmark Collection for Text Categorization Research. J. Mach. Learn. Res.. 5 pp. 361-397 (2004)
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L. & Stoyanov, V. RoBERTa: A Robustly Optimized BERT Pretraining Approach. ArXiv. abs/1907.11692 (2019)
Masala, M., Ruseti, S. & Dascalu, M. RoBERT - A Romanian BERT Model. International Conference On Computational Linguistics. (2020)
https://doi.org/10.18653/v1/2020.coling-main.581
Muffo, M. & Bertino, E. BERTino: An Italian DistilBERT model. Italian Conference On Computational Linguistics. (2020)
https://doi.org/10.4000/books.aaccademia.8748
Nicolae D. C., Tufis D. RoITD: Romanian IT Question Answering Dataset. ConsILR-2021. (2021)
Niculescu, M., Ruseti, S. & Dascalu, M. RoGPT2: Romanian GPT2 for Text Generation. 2021 IEEE 33rd International Conference On Tools With Artificial Intelligence (ICTAI). pp. 1154-1161 (2021)
https://doi.org/10.1109/ICTAI52525.2021.00183
Park, C., Song, H. & Lee, C. S 3 -NET: SRU-Based Sentence and Self-Matching Networks for Machine Reading Comprehension. (2020)
Radford, A. & Narasimhan, K. Improving Language Understanding by Generative Pre- Training. (2018)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. & Sutskever, I. Language Models are Unsupervised Multitask Learners. (2019)
Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. Proceedings Of The 2016 Conference On Empirical Methods In Natural Language Processing. pp. 2383-2392 (2016,11), https://aclanthology.org/D16-1264
https://doi.org/10.18653/v1/D16-1264
Richardson, M., Burges, C. & Renshaw, E. MCTest: A Challenge Dataset for the Open- Domain Machine Comprehension of Text. Conference On Empirical Methods In Natural Language Processing. (2013)
Sanh, V., Debut, L., Chaumond, J. & Wolf, T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv. abs/1910.01108 (2019)
Sarzyńska-Wawer, J.,Wawer, A., Pawlak, A., Szymanowska, J., Stefaniak, I., Jarkiewicz, M. & Okruszek, L. Detecting formal thought disorder by deep contextualized word representations. Psychiatry Research. 304 (2021)
https://doi.org/10.1016/j.psychres.2021.114135
Scheible, R., Thomczyk, F., Tippmann, P., Jaravine, V. & Boeker, M. GottBERT: a pure German Language Model. ArXiv. abs/2012.02110 (2020)
Schwartz, R., Dodge, J., Smith, N. & Etzioni, O. Green AI. Communications Of The ACM. 63 pp. 54 - 63 (2019)
https://doi.org/10.1145/3381831
Tufis, , D. Romanian Language Technology - a view from an academic perspective. INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS and CONTROL. 17 (2022,1), https://doi.org/10.15837/ijccc.2022.1.4641
https://doi.org/10.15837/ijccc.2022.1.4641
Tufis D., Filip F. G. (coordinators). Limba Romana in Societatea Informationala - Societatea Cunoasterii, editura Expert . ISBN: 973-8177-83-9 pp. 512 (2002)
Schwenk, H. & Li, X. A Corpus for Multilingual Document Classification in Eight Languages. ArXiv. abs/1805.09821 (2018)
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, Ł. & Polosukhin, I. Attention is All You Need. Proceedings Of The 31st International Conference On Neural Information Processing Systems. pp. 6000-6010 (2017)
Vries, W., Cranenburgh, A., Bisazza, A., Caselli, T., Noord, G. & Nissim, M. BERTje: A Dutch BERT Model. ArXiv. abs/1912.09582 (2019)
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. & Le, Q. XLNet: Generalized Autoregressive Pretraining for Language Understanding. Proceedings Of The 33rd International Conference On Neural Information Processing Systems. (2019)
Zadeh L., Tufis D., Filip F.G., Dzitac I.(editors) From Natural Language to Soft Computing: New Paradigms in Artificial Intelligence, Editura Academiei . ISBN: 978-973-27-1678-6 pp. 268 (2009)
Additional Files
Published
Issue
Section
License
Copyright (c) 2023 Constantin Dragos Nicolae, Rohan Kumar Yadav, Dan Tufiş
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
ONLINE OPEN ACCES: Acces to full text of each article and each issue are allowed for free in respect of Attribution-NonCommercial 4.0 International (CC BY-NC 4.0.
You are free to:
-Share: copy and redistribute the material in any medium or format;
-Adapt: remix, transform, and build upon the material.
The licensor cannot revoke these freedoms as long as you follow the license terms.
DISCLAIMER: The author(s) of each article appearing in International Journal of Computers Communications & Control is/are solely responsible for the content thereof; the publication of an article shall not constitute or be deemed to constitute any representation by the Editors or Agora University Press that the data presented therein are original, correct or sufficient to support the conclusions reached or that the experiment design or methodology is adequate.
