Analysing student engagement with large language models in higher education: prompts as channels of communication with AI

Thesis

Background: The aim of this study is to establish concrete insights into the ways in which students in higher education are engaging with large language models (LLM) through an analysis of the prompts they have written. Integrating frameworks from Human-Machine Communication (HMC) studies and student engagement literature, this research intends on elucidating prompts as a valuable source of analysis for the effect that LLMs have on students’ approaches to their academic work. These findings will inform scholarly perspectives on the precise strategies, participations and mannerisms students employ within their communications with LLMs.

Method: This research utilises a questionnaire to collect samples of conversations and individual prompts from students' interaction with LLMs in academic contexts. It also asks for their academic background, such as what programme they enrolled in and previous areas of academic expertise. It analyses over 800 prompts from 23 students in higher education within the United Kingdom.

Analysis of this data employed an integrative approach, relating Guzman and Lewis’s (2022) framework for human-AI communications with dimensions of student engagement. This approach enables the analysis of prompt data not only within its specific context as a unique mode of communication with a technological entity but also sheds light on the pedagogical implications of these interactions.

Results: The results from this study presents four main types of engagement with large language models from students in higher education: linguistic manipulation, engagement with corpus data, controlling the LLM and ideation. These engagements reveal how students are mostly benefitting from the LLM’s ability to reword large bodies of text into different styles of language, as well as the ability to navigate the corpus data as a resource for information. Students also utilised LLMs to generate responses that would assist with the exploration of new ideas, as well as the evaluation of ideas. A large portion of prompts were also employed to control the LLM, directing the LLM to respond in certain ways or attempting to fix the LLM’s interpretation of previous prompts.

Authors: Xu, E

• DOI: 10.5287/ora-y5byva9qa
• Type of award: MSc taught course
• Level of award: Masters
• Awarding institution: University of Oxford
• Language: English
• Keywords: AI; LLM; DSC; DSC2024; prompts; human-machine communication; student engagement; artificial intelligence; large language model; questionnaire
• Subjects: Artificial intelligence; Education; Generative artificial intelligence