Reconsidering Writing Pedagogy in the Era of ChatGPT: Results of a Usability Study of ChatGPT in Academic Writing

Lee-Ann Kastman Breuch University of Minnesota
Asmita Ghimire University of Minnesota-Twin Cities
Kathleen Bolander University of Minnesota-Twin Cities
Stuart Deets University of Minnesota-Twin Cities
Alison Obright University of Minnesota-Twin Cities
Jessica Remcheck University of Minnesota-Twin Cities

Introduction

The emergence of Generative Artificial Intelligence (AI) has produced both havoc and hope. Scholars in the field of composition have suggested that ChatGPT–a Generative AI tool—has changed the dynamics of teaching, writing and teaching writing (Baidoo-Anu & Owusu, 2023; Floridi, 2023). A recent statement by a combined MLA-CCCC task force suggested that ChatGPT would affect the creative and critical thinking of students, add more labor to teachers and instructors without compensation, and would challenge the institutional policy and practices by asking it to redefine the idea of plagiarism. Composition scholars are also raising questions about the changing dynamics of writing, literacy and pedagogical practices. For example, Johnson (2023) suggested that we redirect the threshold concepts of writing to tackle the present changing dynamics of teaching writing. Anson and Straume (2022) similarly encouraged instructors to work with AI tools rather than “combat or extinguish” them (p. 7).

In this article, we aim to extend scholarly discussions of generative AI tools by sharing results of a usability study of ChatGPT involving undergraduate students at a large midwestern university. We set out to learn how students were understanding ChatGPT texts and how they might imagine using ChatGPT texts for academic purposes. Thirty-two student participants completed five academic writing tasks using ChatGPT, rated the ChatGPT texts, and answered debriefing questions. As our findings show, students had a rhetorical and multidimensional response to ChatGPT, acknowledging its impressive capabilities but also raising key questions about using ChatGPT in several areas. Ratings showed that students were mostly satisfied with ChatGPT but were less confident about using ChatGPT texts unaltered as their academic homework. Students raised questions about the credibility of ChatGPT texts, uses of sources, integration of citations, tone and voice of ChatGPT texts, and how use of the text would impact their sense of scholarly identity and quality of learning. In sum, students expressed rhetorical critiques of ChatGPT while also sharing their impressions, both positive and negative.

Based on qualitative and quantitative findings, we propose pedagogical principles that would help instructors and students to engage the generative AI in writing and classrooms. The principles include expanding writing pedagogy to integrate generative AI in writing processes, developing critical AI literacy, and addressing ways that generative AI impacts textual integrity and plagiarism. We also provide screenshots and videos of ChatGPT sessions and highlight student comments to share perspectives of what students valued and questioned about ChatGPT.

Next Topic

Introduction

The Rise of AI and ChatGPT

Study Methods

Results

Qualitative Student Comments about ChatGPT

Discussion

Pedagogical Implications

Conclusion

References

Other chapter(s) in this category

Reports from the Field: Classes & Students Using AI

Make it So: What Students at a Large Public University Really Think about Generative AI

By Jeanne Law-Bohannon, John C. Havard, and Laura Palmer

ChatGPT is Not Your Friend: The Importance of AI Literacy for Inclusive Writing Pedagogy

By Mark C. Marino

About the authors

Lee-Ann Kastman Breuch

Lee-Ann Kastman Breuch, Ph.D., is Professor in the Department of Writing Studies at the University of Minnesota and Associate Dean for Undergraduate Education in the College of Liberal Arts. Her research investigates rhetoric and digital writing in a variety of settings such as classrooms, professional organizations, and social media. She teaches courses in technical communication, digital writing, usability research, and evaluation of online interfaces.

Asmita Ghimire

Asmita Ghimire is a PhD candidate in the Rhetoric, Scientific and Technical Writing program at the University of Minnesota–Twin Cities. Her research areas are Rhetoric, International and Intercultural Technical Communication, and Transnational Feminist Studies.

Kathleen Bolander

Kathleen Bolander is a PhD student in the Department of Writing Studies at the University of Minnesota–Twin Cities. She is also a graduate mentor with UMN Athletics. Her main area of research is digital rhetoric and the rhetoric of silence.

Stuart Deets

Stuart Deets is a PhD student in the Department of Writing Studies at the University of Minnesota–Twin Cities. His research focuses on communicating public policy.

Alison Obright

Alison Obright is a PhD student in the Department of Writing Studies at the University of Minnesota - Twin Cities. Her research focuses on the rhetoric of science, pseudoscience, and technology.

Jessica Remcheck

Jessica Remcheck is a PhD student in the Department of Writing Studies at the University of Minnesota–Twin Cities. Her research areas include rhetoric of health and medicine and rhetoric of science.