Evaluasi Kualitas, Akurasi, dan Stabilitas Penilaian Pemrograman Berbantuan Model Bahasa Besar
DOI:
https://doi.org/10.59188/jurnalsostech.v6i10.33003Keywords:
Bias Sistematis, ICC, LLM, Penilaian Otomatis, Praktikum PemrogramanAbstract
Model bahasa besar (LLM) menawarkan peluang otomatisasi penilaian pemrograman, tetapi kelayakannya memerlukan bukti akurasi nilai, stabilitas, dan kualitas umpan balik. Penelitian ini bertujuan mengevaluasi ketiga dimensi tersebut menggunakan data empat kelas praktikum pemrograman yang melibatkan 197 mahasiswa dan 4.664 jawaban. Pendekatan Design Science Research digunakan untuk pengembangan dan evaluasi. Akurasi diperiksa melalui 163 pasangan nilai dengan penilaian asisten historis sebagai pembanding menggunakan korelasi, metrik galat, analisis bias, kesepakatan Bland–Altman, dan kesalahan batas. Stabilitas diuji melalui penilaian berulang dan perbandingan konfigurasi, sedangkan kualitas umpan balik dinilai menggunakan ukuran tekstual otomatis dan tiga penilai LLM. Hasil menunjukkan korelasi kuat dengan nilai asisten (r = 0,854), tetapi disertai bias negatif sistematis 8,97 poin, MAE 10,09, dan RMSE 13,39. Kesalahan klasifikasi lulus menjadi gagal mencapai 17,2–18,4% pada ambang yang diperiksa. Penilaian berulang menunjukkan konsistensi tinggi (ICC = 0,9943), tetapi perubahan versi penilai menggeser nilai 2,49 poin dan pengaturan temperature nol meningkatkan variasi 4,7 kali. Umpan balik memperoleh penilaian kebenaran faktual dan kebahasaan lebih tinggi daripada keterlaksanaan saran, yang memiliki rerata 2,633. Temuan ini menunjukkan bahwa konsistensi belum menjamin validitas penilaian. LLM dapat mendukung pemeringkatan dan penyaringan, tetapi penetapan nilai akhir memerlukan kalibrasi, pengendalian versi, serta pengawasan manusia, terutama pada keputusan akademik dekat ambang kelulusan.
References
AlGhamdi, E. M., Li, Y., Gašević, D., & Chen, G. (2026). Leveraging prompt-based LLMs for automated scoring and feedback generation in higher education. Computers & Education, 243, 105511. https://doi.org/10.1016/j.compedu.2025.105511
Cisneros-González, J., Gordo-Herrera, N., Barcia-Santos, I., & Sánchez-Soriano, J. (2025). JorGPT: Instructor-aided grading of programming assignments with large language models (LLMs). Future Internet, 17(6), 265.
Evaluating the capability of large language models in characterising relational feedback: A comparative analysis of prompting strategies. (2025). Computers and Education: Artificial Intelligence, 8, 100427. https://doi.org/10.1016/j.caeai.2025.100427
Gerke, O. (2020). Reporting standards for a Bland--Altman agreement analysis: A review of methodological reviews. Diagnostics, 10(5), 334.
Gerke, O., & Möller, S. (2021). Bland--Altman limits of agreement from a Bayesian and frequentist perspective. Stats, 4(4), 1080–1090.
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., & others. (2026a). A survey on llm-as-a-judge. The Innovation, 7(6).
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., & others. (2026b). A survey on llm-as-a-judge. The Innovation, 7(6).
Han, S. S., Takashima, Y., Shen, S. Z., Liu, C., Liu, Y., Thuo, R. K., Knowlton, S., Piskac, R., Shapiro, S. J., & Cohan, A. (2025). COURTREASONER: Can LLM Agents Reason Like Judges? Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 35279–35294.
Jukiewicz, M. (2026a). A systematic comparison of large language models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor. Computers and Education Open, 10, 100364.
Jukiewicz, M. (2026b). A systematic comparison of large language models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor. Computers and Education Open, 10, 100364.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163.
Messer, M., Brown, N. C. C., Kölling, M., & Shi, M. (2025). How consistent are humans when grading programming assignments? ACM Transactions on Computing Education, 25(4), 1–37.
Norman, J. D., Rivera, M. U., & Hughes, D. A. (2026). Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias. ArXiv Preprint ArXiv:2606.19544.
Raj, J. N., Muppa, A., Nirmal, R., Kamath, T. W., Boicu, M., Dipukumar, A., & Laddha, A. (2025a). BOARD# 97: WIP: The Effectiveness of Rubric-Based LLM Feedback for Programming Assessments. 2025 ASEE Annual Conference & Exposition.
Raj, J. N., Muppa, A., Nirmal, R., Kamath, T. W., Boicu, M., Dipukumar, A., & Laddha, A. (2025b). BOARD# 97: WIP: The Effectiveness of Rubric-Based LLM Feedback for Programming Assessments. 2025 ASEE Annual Conference & Exposition.
Vom Brocke, J., Hevner, A., & Maedche, A. (2020). Introduction to design science research. In Design science research. Cases (pp. 1–13). Springer.
Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 3087.
Yagubyan, A. (2026). The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation. ArXiv Preprint ArXiv:2606.13685.
Yousef, M., Mohamed, K., Medhat, W., Mohamed, E. H., Khoriba, G., & Arafa, T. (2025). BeGrading: large language models for enhanced feedback in programming education. Neural Computing and Applications, 37(2), 1027–1040.
Zapf, A., Castell, S., Morawietz, L., & Karch, A. (2016). Measuring inter-rater reliability for nominal data--which coefficients and confidence intervals are appropriate? BMC Medical Research Methodology, 16(1), 93.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Naufal Raihan Saputra, Ahmad Faqod Kurnia, Jenny Agustina Rahman
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA). that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
