The Performance of Large Language Models in Synthesising Educational Research Reports: A Comparative Study of Articles in Portuguese

Abstract

This study compares the performance of ChatGPT 5.0, Gemini 3.0 Flash, and DeepSeek V3 in producing abstracts of scientific articles published in Portuguese. A mixed-methods approach was adopted, involving the submission of five articles in the field of Education to the three language models using three prompt variations. The outputs were evaluated by 15 specialists, with an ordinal perception of quality (PQ) scale to assess the dimensions of content quality and freedom from bias. The results indicated that the models performed well, with PQ medians ranging from 84% to 94% of the total score. The Kruskal–Wallis test identified statistically significant differences when the results were compared by article, H(4)=24.040,p<0.001, but found no significant differences among the models, H(2)=0.631,p=0.729, or among the prompt types, H(2)=1.604,p=0.448. These findings suggest a technological convergence among the models in the task of scientific text synthesis. However, the qualitative analysis identified  generalizations’ tendency and the suppression of uncertainty markers, indicating shortcomings in the validity of the generated abstracts. The findings also suggest the possibility of biases arising from the languages represented in the models’ training data, which could hypothetically affect their ability to capture the narrative texture and specialized terminology of scientific knowledge produced in Portuguese. It was concluded that, although the generative AI models tested demonstrated good syntactic performance, human oversight remains indispensable for ensuring the accuracy and completeness of scientific communication in educational research.