Composing with AI

The Turing Test

Alan Turing, father of computing, presented the world with an imitation game, or rather two. In the first, a man tries to pass himself off as a woman. In another, analogous game, a computer tries to pass itself off as a man, for a man could hardly calculate with the speed of a computer. He offers these games as alternatives to the tarpit created by the question of machine intelligence. By the end of the century, he claimed, computers would be able to pass themselves off as humans in limited situations more than half the time. ChatGPT goes a long way to achieving that goal. But Turing’s game inspired an in-class activity.

In this exercise, students answer a reflection question in four sentences. I like choosing a question about the nature or future of machine intelligence for the echoes with Turing. Then, they generate another four-sentence answer from ChatGPT or other LLM. The students post one of the two answers in a discussion forum, and then the class tries to guess whether the text was human or machine-generated. The results surprised me.

Even in our first run of this exercise, the students produced text that was quite difficult to identify as either human or machine-generated. On the one hand, students were fairly sophisticated in their prompting, cuing the machine to write as a middle or high school student or to make some errors. On the other hand, and this surprised me even more, students had developed some skill at imitating the cadence and rhythm of ChatGPT, especially when lightly prompted. They had developed the ability to imitate what I call its person-less prose, consisting of complete sentences of similar length and structure.

Thus, rather than testing the AI’s performance, this exercise tested the students’ skill at producing language in voices other than their own. That dexterity with word choice, that ability to recognize and imitate style is something that I would happily include in the learning outcomes for my class. Furthermore, the exercise suggested a relationship to prompting chatgpt that far more sophisticated than merely asking for text and submitting it as their own, the scenario most alarmist offer as the harbinger of the end times of composition instruction. By contrast, this exercise showed how the AI could be a central part of a discussion of what makes human writing unique and how LLMs munge their input training sets.