Can Large Language Models deliver human-equivalent essay feedback without prompt-specific pre-training?

Working paper

This study investigates the potential of general-purpose large language models (LLMs) to support essay assessment at scale without the bespoke tuning that currently characterises most AI applications. Using structured prompts and pre-existing analytic rubrics, the research examined whether LLMs could generate scores and formative feedback that align with human judgement in both technical and pedagogical dimensions. Drawing on a dataset of 17,000 student essays across four iterative phases, the study evaluated whether prompt engineering alone, without task-specific pretraining, can achieve levels of scoring accuracy and feedback quality comparable to those of trained educators. Findings demonstrate that, within defined conditions, general-purpose models can attain human-equivalent performance. The standard model improved from initial inconsistency to the level of an average human grader, while the advanced model achieved consistent expert-level agreement. These results were validated across multiple genres, grade levels, and rubric structures using psychometric indicators including Quadratic Weighted Kappa (QWK), Intraclass Correlation Coefficient (ICC), and Root Mean Squared Error (RMSE). Notably, the models performed strongly on both mechanical and higher-order traits, such as comprehension and elaboration. The study concludes that, when carefully designed and pedagogically contextualised, LLMs can enhance the scalability, consistency, and responsiveness of formative assessment without displacing human interpretive judgement.

Authors: Ratner, S; Lynch, M; Cooney, G

• Publication status: Published
• Publisher: University of Oxford
• Place of publication: Oxford, UK
• DOI: 10.5287/ora-pdgy6dgxv
• Language: English
• Keywords: marking; assessment; LLM; AI; essay feedback