A genre, scoring and authorship analysis of AI-generated and human-written business refusal emails: is there still a case for business writing training?

Thesis

In a business context, writing negative messages such as refusals to requests is particularly challenging, as the act of refusal can damage the relationship between writer and reader. Generative artificial intelligence (AI) tools have shown the capacity to assist with various writing tasks, but do these tools remove the need for business writing training? This study analysed the genre moves and steps that typify business refusal texts generated by ChatGPT (n = 18) and Gemini (n = 18) alongside human-written texts (n = 18). To assess writing quality, experienced business communication instructors (N = 36) scored the texts on an established rubric for evaluating business written professional communication. Assessors also made authorship judgements as to whether each text was AI-generated or human-written and provided rationales for their decisions. The views of the instructors regarding the use of AI tools for writing in the workplace were also captured.

The genre analysis showed that AI-generated texts were formulaic and largely followed the same moves and steps, regardless of text type or audience, whereas human-written texts were more nuanced and flexible in their adoption of genre conventions. On writing quality, human-written texts achieved the highest mean scores, followed by Gemini texts and finally ChatGPT. Inferential statistics revealed that Gemini texts did not differ significantly in the rubric evaluation scores compared to human-written texts, although Chat-GPT produced texts received significantly lower evaluations of writing quality. Participants were able to identify authorship of AI-generated texts with an accuracy rate of 68.1%, and human-written texts were correctly identified 86% of the time. Qualitative data revealed that assessors were most concerned with three broad characteristics of the texts: tone, relationship and formality; language choice and grammatical accuracy; content, detail and structure. Insights into key differences between AI-generated texts and human writing drawn from these results are used to inform five key areas of focus for teaching business writing in the AI age.

Authors: Wilson, W

• DOI: 10.5287/ora-a4a9g9mog
• Type of award: MSc taught course
• Level of award: Masters
• Awarding institution: University of Oxford
• Language: English
• Keywords: Gemini; ALLT; ALLT2024; genre analysis; AI; business refusals; business emails; authorship analysis; scoring analysis; business writing; ChatGPT
• Subjects: Applied linguistics; Artificial intelligence; Authorship; Business