Composing with AI

Classroom Heuristic Development Methodology

We engaged with GPT-3.5 interface via https://chat.openai.com/. Presently, this is the same website interface for GPT-4, which is the paid tier I subscribed to and engaged with along with GPT-3.5. I provided students with a survey before and after they engaged with ChatGPT on their own computers through small group, semi-structured activities over two class periods. In writing up the analysis, I’ve integrated my discussion with aggregate/paraphrased student responses. In terms of demographics, students were traditional undergraduates, primarily from small town, rural, and coastal Eastern North Carolina with a few students from more urban Southeast or Northeast states and one international student from Mexico. Most students identified as female, although there were male and non-binary students. Most students identified as white with about 1/3 of the classroom identifying as Black, Latinx, or by one or more non-white ethnicities.

In the survey, I asked a range of questions about students’ writing processes and their knowledge of ChatGPT. Just over half of students considered themselves good writers (57.1%) with slightly more students claiming to enjoy writing (61.9%). When asked to describe their writing in three words, many students used values of simplicity, efficiency, safety, logic, organization, and clarity to describe the current state of their writing or to the kinds of writing skills they’d like to improve. These values echoed the “professionalism” Selfe and Selfe observed as white, middle-class values and demonstrated that college writers use those values to identify standards of good writing for their own written products. Of the positive words students associate with their own writing, responses included descriptions such as creative, enlightening, appealing, insightful, well-spoken, feminine, dark, expansive, bright, fun, and unique. These positive descriptors were more likely to appear individually and attached to a particular student while the professionalism values were more prevalent through the responses. When asked to identify their greatest strengths as writers, students chose creativity (13), organization (13) and originality (12), and they believed using correct grammar/spelling (11), incorporating and citing research (11), and doing a final proofread (10) were their greatest weaknesses. When asked about aspects of their writing process they engaged in for school or workplace purposes, most students said they brainstormed (12), performed preliminary research (15), drafted (13), and sought out feedback from a professor, tutor, or friend/family member (16). Interestingly, far fewer students claimed to engage in revising (7) and incorporating and citing research (4) as part of their usual writing process for school or work.

In the class I surveyed, 81% of students stated they were aware of ChatGPT and 19% of students stated they’d never heard of it. When asked to associate three words with ChatGPT, responses expressed polemical attitudes: “helpful” was used as many times as “cheating” with terms like “robotic,” “answers,” “easy,” and “AI” circulating along with “awesome” and “evil.” When asked to explain what ChatGPT does, most students stated some version of “it gives you answers,” “it knows everything,” “it helps you with assignments,” and “it writes your paper for you” or they stated they didn’t know what it did (4). When asked to describe how it works, students either indicated they didn’t know or that it worked “through AI” with a few students mentioning more granular attributes like “software,” “metadata and analysis,” and process like “uses an algorithm” and “has a lot of data it has learned from the internet.” When asked to identify the feelings they have about ChatGPT, students most frequently chose excited (10), intimidated (9), distrustful (9), and optimistic (8) with over half of the students stating they’d used ChatGPT and been satisfied with the results. When asked what had satisfied them, students commented on ChatGPT’s ability to provide useful outlines, to help them learn things like cooking and shaving, and to answer all the questions they might have about school, work, and life.

Beyond the content of the responses, students were most broadly satisfied by ChatGPT’s speed and efficiency—they responded most positively to its performance--its instantaneous, authoritative responses seemingly typed across their person screens. When I asked students who hadn’t used ChatGPT their primary reason why, most chose, “I’m concerned about the consequences” (4) with some students not knowing how to try it (2) or being concerned specifically about teachers/school frowning upon it (2). Only 23.8% (5 of 24) of students admitted to using ChatGPT to write something they used for school or work with those students split on feeling positively or negatively about the experience. Half of students said they’d like professors to teach them how to use ChatGPT (52.4%) with 28.6% opposed to ChatGPT instruction from their teachers and ~20% ambivalent or conflicted about classroom use and instruction. When asked if ChatGPT could write better than most people, 57% of students thought it couldn’t write better than most people but 52.4% of students thought it could write better than they could. When asked about ethical concerns, 76% of students said they had concerns mostly related to cheating or misrepresenting intellectual property with some darker responses about AI/tech being scary and/or dangerous to humanity. A few responses evoked concern over risks to their learning and intellectual/creative growth. When asked if using ChatGPT had no negative consequences, if they would prefer to use ChatGPT or another similar technology to do most or all their writing, 61.9% said no, while 38.1% answered positively. Students who indicated they wouldn’t use ChatGPT regularly said that they valued their own writing pleasure, originality, creativity, learning, and humanity over ChatGPT’s functions. The smaller group who said they would use ChatGPT for most writing remarked on its efficiency, ease of use, clarity, or their own perceived flaws, like “laziness” or lack of understanding, as well as the cultural imperative to seize an innovative technology for personal use. Finally, and maybe most pertinently, most students believed in the next five years it would become necessary to know how to work with generative AI like ChatGPT in college (76.2%) and in most professional environments that required writing (81%). Although the sample for this survey is small, it gave me a valuable glimpse into student attitudes, beliefs, and values around writing amid a technological paradigm shift in personal writing practices and writing studies research. This glimpse made students’ vulnerability to this particular technological instantiation acutely visible. ChatGPT seems to have arrived just for them—it answers all their questions without judgement and provides “perfect answers” that may help them forge successful personal and professional paths forward in their lives in which. Perhaps most sobering, no matter their personal view on tech like ChatGPT, they believe they’ll ultimately be compelled to use it in school or work contexts.

Prior to surveying students, we refrained from discussing ChatGPT, and I set aside two class period as ChatGPT workshops. I explained the research scenario for this book chapter, telling them that I’d like their participation and feedback, but that feedback would be anonymous, and they could opt out of any aspect at any time. The first survey elicited 21 of 24 student responses. According to my observations, most to all of students participated in the in-class activities around ChatGPT. Due to time constraints in our semester, I didn’t administer the second survey in class, and only 8 of 24 students provided responses. If given another opportunity, I’d provide another week to complete the sequence of survey, activities, and discussion to provide more in-class time for heuristic development and analysis.

Heuristic Discussion

As part our in-class activities, I asked students to watch OpenAI’s March 15, 2023 promotional video. This 3-minute video is a masterclass of rhetorical maneuvers. Towards the end, a woman says, “We think that GPT-4 will be the world’s first experience with a highly capable and advanced AI system. So, we really care about this model being useful to everyone, not just the early adopters or people very close to technology.” In this moment, OpenAI explicitly hails the public—all of us are the ideal user for GPT—no matter our technical knowledge or skills. Further attempting to establish an ethic of care about the technology’s development, its use, and its effect on our lives, the woman says, “So it is really important to us that as many people as possible participate so that we can learn more about how it can be helpful to everyone.” With this, OpenAI appeals to commonplace cultural values of being helpful and contributing to the public good as the reason we should use ChatGPT.

When students were asked about their impressions of the promotional video, responses were mostly positive. Students felt as though the video provided more information about ChatGPT’s capabilities, its improvements, and, perhaps most notably, its ability to learn from them while also serving as their personal tutor. Regarding interface design, most students indicated they liked its simplicity and ease of use, the speed of responses, the absence of additional links to click for information, the form of interaction because it resembles texting someone, the visual streaming response, and the archive ability for older chats. When asked what they disliked about the interface, nearly half of students said there wasn’t anything about the interface they didn’t like (3), while others stated it was too basic/bland/simple (4), or its responses took too long to generate (1). When asked what they would change about GPT’s interface design or functionality to better reflect their preferences and desires, most of the responding students (5) stated they would not change any aspects of its interface design or functionality. The other half responded with ideas to change the re-generate feature and memory for better storytelling (1), an option for more up-to-date information (1), a function to determine trustworthy information (1), and, lastly, “a design that will appeal to the outside audience (1).” This last response is one that I’d like to pursue.

What did the student mean by “outside audience”? How did the students in front of me oriented themselves to the bodies, identities, and values presented in OpenAI’s initial ChatGPT promo video? This student’s response spoke to a a wider perspective on ChatGPT and growing critical awareness of its norming potential on “outside audiences”. In our class activity, we didn’t quite have the time we needed to fully develop guiding principles to heuristically analyze ChatGPT’s interface, but student perspectives demonstrated vulnerability to AI’s logics and values that need to be addressed to ensure we continue fostering students having agency over their writing through critical evaluation and remaking of generative writing tool interfaces.