The advent of Large Language Models (LLMs) has ignited extensive discussion surrounding the impact of generative artificial intelligence (GenAI) on education and literacy practice (such as the process and outcomes of reading and writing) (see for example Altmäe et al., 2023; Dans, 2023). Assessment is at the core of curriculum design and often is regarded by students as the driver of learning. Reimagining assessment could drive the transformation of education in response to the impact of GenAI on education and post-graduation professional practice.

Higher education professionals, while marvelling at the human-like responses of GenAIs to essay-based topics, express grave concerns about the risks that GenAI poses to essay-based assessment. The difficulty in discerning the authorship of GenAI-assisted essays has led to calls for a return to traditional paper and in-person examinations to uphold academic integrity (Lo, 2023). Yet, throughout the history of Educational Technology (EdTech), there was ‘frequently insufficient consideration for how educators implemented, and students interacted, with such resources’ (Rudolph et al., 2023, p. 2). Consequently, the purported radical innovation in higher education following the introduction of new technologies is frequently overstated and often not substantiated by thorough research.

‘The purported radical innovation in higher education following the introduction of new technologies is frequently overstated and often not substantiated by thorough research.’

The integration of GenAIs into literacy practices is unstoppable. To fulfil our missions as educators – nurturing a workforce for the 21st century empowered by GenAI – we must shift from questioning whether GenAI should be used in assessment to how we can transform our assessment practice. This transformation could encourage students to cultivate skills that enable them to collaborate and engage productively with GenAIs.

Redefining essay-based assessment to address changes brought about by GenAIs hinges on meticulous assessment design, significantly shaped by the assessment literacy of both designers and assessors. A vital component of assessment literacy to address the repercussions of GenAIs on assessment constructs is understanding the distinctions between ChatGPT-assisted and human-only writing.

Our ongoing British Council-funded project seeks to investigate GenAI’s effects on the processes and outcomes of disciplinary writing. Our presentation at the forthcoming BERA/WERA conference 2024 will focus on the impact of GenAIs on writing. Specifically, we aim to answer the questions:

1. How does ChatGPT-assisted university writing differ from university student writing when based on the same assessment tasks and assessment criteria?
2. Which assessment constructs should be retained or eliminated when assessing GenAI-assisted writing?

Responding to these questions, we scrutinised the differences between ChatGPT-assisted and human-only student writing by performing corpus analyses of: (a) writing produced solely by humans and those with ChatGPT assistance; and (b) disciplinary lecturers’ feedback on these scripts. Sixty students engaged with ChatGPT to assist with their written coursework. Disciplinary lecturers employed the same assessment rubrics used in previous years for solely human writing to evaluate the knowledge, argumentation and effective communication in ChatGPT-assisted writing.

At the conference, we will present our findings on the unique characteristics of human-only versus ChatGPT-assisted writing from the perspective of subject lecturers. We will discuss how to redefine assessment tasks and rubrics to tackle essential skills for conveying subject knowledge in disciplinary writing, in partnership with GenAIs. Additionally, we will discuss how these results could inform the identification of authorship in GenAI-assisted disciplinary writing.

Our ways forward

As educators, rather than refuting AI as a disruptive technology, we should/might consider how GenAI can be harnessed for positive societal outcomes. To this end, we need to cultivate a constructive technological culture (Ellul, 1990) that promotes:

• the acquisition of technological knowledge of using AI for learning and assessment
• the adaptation of students and professionals to the AI-powered technological environment to make them comfortable in it
• the creation of a psychological mood that is favourable to the technique, and openness to everything about it.

To catalyse this cultural shift, rigorous research in partnership with our students is paramount. We invite colleagues and students to join us to share their experiences, perspectives and, more importantly, empirical evidence openly as a community. Together, we can transform our practice to support the community and our students in developing the literacy skills necessary to retain writer agency and, consequently, our creativity and innovation in an AI-powered world.

This blog post relates to a paper presented at the BERA Conference 2024 and WERA Focal Meeting on Tuesday 10 September at 9:00am. Find out more by searching the conference programme here.

References

Altmäe, S., Sola-Leyva, A., & Salumets, A. (2023). Artificial intelligence in scientific writing: A friend or a foe? Reproductive Biomedicine Online, 47(1), 3–9. https://doi.org/10.1016/j.rbmo.2023.04.009  

Dans, E. (2023, January 27). ChatGPT and the decline of critical thinking. https://www.ie.edu/insights/articles/chatgpt-and-the-decline-of-critical-thinking/

Ellul, J. (1990). The technological bluff. W.B. Eerdmans.

Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410   

Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.9