The Quality of Scientific Papers of Undergraduate Students in Regional Language and Literature Studies Lectures Before and After Using ChatGPT as a Writing Tool

Ari Wulandari; Arif Budiman; Tri Ratna Herawati; Primasari Wahyuni; Diaz Danu Setyawan

doi:10.24903/sj.v11i2.2353

Authors

Ari Wulandari Universitas PGRI Yogyakarta
Arif Budiman UIN Sunan Kalijaga Yogyakarta
Tri Ratna Herawati Universitas PGRI Yogyakarta, Indonesia
Primasari Wahyuni Universitas PGRI Yogyakarta
Diaz Danu Setyawan Universitas PGRI Yogyakarta

DOI:

https://doi.org/10.24903/sj.v11i2.2353

Keywords:

ChatGPT, Scientific Papers, Undergraduate Students, Regional Language and Literature Studies, Quality of Academic Writing

Abstract

Background:

This study addresses the growing use of ChatGPT in academic writing. Grounded in constructivism, the study investigates changes in the quality of scientific papers produced by undergraduate students in the Regional Language and Literature Studies course at University “A” before and after the use of ChatGPT as a writing support tool.

Methodology:

A quasi-experimental pretest–posttest design using document analysis was applied. The data consisted of 30 student papers written before and after ChatGPT use. Assessment covered five aspects: ideas, organization, grammar, diction, and spelling.

Findings:

The findings indicate substantial improvements in students’ scientific papers after the use of ChatGPT. The mean writing score increased from 77.9 before ChatGPT-assisted revision to 89.0 after revision, and 27 of the 30 papers improved to the excellent category. The strongest improvements were observed in organization, grammar, diction, and spelling, whereas gains in idea development were comparatively more modest.

Conclusion:

The findings suggest that ChatGPT-assisted revision can enhance the technical quality of students’ scientific writing. However, lecturer guidance remains necessary to support critical thinking, independent analysis, and the responsible use of generative AI in academic writing.

Originality:

This study contributes empirical evidence on changes in the quality of students’ scientific writing associated with ChatGPT-assisted revision within the context an undergraduate language and literature course.

References

Ary, D., Jacobs, L. C., Irvine, C. K. S., & Walker, D. (2021). Introduction to research in education (10th ed.). Cengage Learning.

Barkaoui, K. (2023). Examining the impact of automated writing evaluation on second language writing: A systematic review. Assessing Writing, 55, 100685.

https://doi.org/10.1016/j.asw.2022.100685

Bearman, M., & Ajjawi, R. (2023). AI and the future of assessment in higher education. Medical Education, 57(11), 987–989. https://doi.org/10.1111/medu.15145

Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–623. https://doi.org/10.1145/3442188.3445922

Bowen, G. A. (2009). Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27–40. https://doi.org/10.3316/QRJ0902027

Bruner, J. (1986). Actual minds, possible worlds. Harvard University Press.

Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148

Creswell, J. W., & Creswell, J. D. (2023). Research design: Qualitative, quantitative, and mixed methods approaches (6th ed.). SAGE Publications.

Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE Publications.

Crompton, H., & Burke, D. (2024). Artificial intelligence in higher education: The state of the field. International Journal of Educational Technology in Higher Education, 21, 22. https://doi.org/10.1186/s41239-024-00439-9

Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., Baabdullah, A. M., Koohang, A., Raghavan, V., Ahuja, M., Albanna, H., Albashrawi, M., Al-Busaidi, K. A., Balakrishnan, J., Barlette, Y., Basu, S., Bose, I., Brooks, L., Buhalis, D., ... Wright, R. (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642

Elosua, P., & Lasagabaster, D. (2025). The role of first language resources in multilingual academic writing. Journal of Second Language Writing, 67, 101170. https://doi.org/10.1016/j.jslw.2024.101170

Facione, P. A. (2020). Critical thinking: What it is and why it counts (7th ed.). Insight Assessment.

Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. E. J. (2024). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846

Hyland, K. (2019). Second language writing (3rd ed.). Cambridge University Press.

https://doi.org/10.1017/9781108509383

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeiffer, F., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., ... Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274

Lea, M. R., & Street, B. V. (1998). Student writing in higher education: An academic literacies approach. Studies in Higher Education, 23(2), 157–172. https://doi.org/10.1080/03075079812331380364

Lim, W. M., Gunasekara, A. N., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2024). Generative AI and the future of education: A systematic review of empirical evidence. International Journal of Educational Technology in Higher Education, 21, Article 20. https://doi.org/10.1186/s41239-024-00455-9

Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.

https://doi.org/10.1037/0003-066X.50.9.741

Miles, M. B., Huberman, A. M., & Saldaña, J. (2014). Qualitative data analysis: A methods sourcebook (3rd ed.). SAGE Publications.

Nurgiyantoro, B. (2016). Penilaian pembelajaran bahasa berbasis kompetensi (2nd ed.). BPFE.

Ranalli, J. (2021). L2 student engagement with automated written corrective feedback: Current status and future directions. Language Teaching Research, 25(5), 730–749.

https://doi.org/10.1177/1362168820934230

Reiser, B. J. (2004). Scaffolding complex learning: The mechanisms of structuring and problematizing student work. Journal of the Learning Sciences, 13(3), 273–304. https://doi.org/10.1207/s15327809jls1303_2

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Houghton Mifflin.

Shi, J., Zhang, Y., Liu, X., & Wang, H. (2025). Academic literacy, information literacy, and university students’ academic writing quality. Journal of Writing Research, 17(1), 45–67.

Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10, 15. https://doi.org/10.1186/s40561-023-00237-x

UNESCO. (2023). Guidance for generative AI in education and research. UNESCO.

https://doi.org/10.54675/EWZM9535

Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.

Wingate, U. (2012). ‘Argument!’ helping students understand what essay writing is about. Journal of English for Academic Purposes, 11(2), 145–154. https://doi.org/10.1016/j.jeap.2011.11.001

Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89–100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x

Zhai, X. (2023). ChatGPT for next generation science learning. Journal of Research in Science Teaching, 60(8), 1524–1530. https://doi.org/10.1002/tea.21816

Zhang, Z. V., & Hyland, K. (2023). Student engagement with automated feedback on academic writing: Revising, responding, and learning. Assessing Writing, 55, 100694.

https://doi.org/10.1016/j.asw.2022.100694