What Students Think about Generative AI
Jeanne Law, James Blakely, John C. Havard, and Laura Palmer
Methodology
Our study team includes faculty, graduate students, and administrators from KSU’s Department of English as well as faculty and administrators from the Department of Technical Communication and Interactive Design (TCID). We are at the beginning of what we hope to be a longitudinal study of students’ attitudes toward generative AI use in writing contexts. We have experienced strong interest across multiple disciplines at KSU, and we plan to categorize research findings along several trajectories. For this particular study, we focused primarily on first-year students enrolled in ENGL 1101 and ENGL 1102. We also performed a snapshot method in a smaller population of students enrolled in a non-General Education Technical Communication course (TCOM 2010). While this snapshot is preliminary in terms of comparison to our target population, we did find interesting trends that we present in our article.
Using the Qualtrics XM survey platform with Text and Stats IQ analysis tools, our team designed an eight-question instrument with both closed and open-ended response options. Our goal was to begin a preliminary measurement of first-year students’ attitudes towards generative artificial intelligence (gen-AI) in their writing writ large, with specific attention paid to gen-AI in their academic lives. We were most interested in their responses towards AI as the future of writing and whether or not they considered gen-AI cheating in their academic writing courses.
Study and Respondent Demographics
Our team surveyed students enrolled in first-year writing courses at Kennesaw State University (KSU) in Fall semester 2023. Kennesaw State University is a public, R2 doctoral-granting school founded in 1963. It comprises two campuses totalling 544 acres. The acceptance rate is 68%. KSU is the second largest university in Georgia, with a total enrollment of 43,268. Of this number, 39,005 are undergraduates.
In Fall 2023, KSU welcomed more than 8,500 students as first-year students, who were enrolled in more than 350 first-year composition (FYC) courses taught by almost 100 faculty. These students were demographically diverse in several ways:
- 38% self-identify as first-generation students
- Approximately 50/50 gender split
- 25% self-identify as Black or African American
- 13% self-identify as Hispanic of Latinx
- 6% self-identify as Asian
- 5% self-identify as two or more races
- 48% self-identify as White
- 79% of undergraduates are full-time students
KSU students are mostly conventional college-age, with 84% of undergraduate students falling between the ages of 19-24. KSU has a robust dual-enrollment program, with more than 800 students.
Survey Validity and Reliability
Our mixed methods survey on first-year students' attitudes toward AI use in writing courses is a reliable and valid instrument for measuring initial attitudes towards generative AI. We designed the survey as a short snapshot that would balance the content validity of collected data with students’ actually taking the time to complete the survey. To this end, we used a 3-point Likert-like scale of yes/no/sometimes in our closed questions, with follow-ups to these questions that were open-ended, asking students to explain their answers in more detail. Our survey sample as of October 15, 2023, was 942 respondents, which represents 11% of first-year enrollment. We continue to make this survey available through a systematic distribution that entails the Department Chair (John) and the Director of First-Year Composition (Jeanne) sending the survey link to all instructors in the First-Year Composition (FYC) Program encouraging them to send the link to their students via email, LMS post, or classroom time. We also send email reminders during the semester and promote the survey at monthly Department and FYC meetings. Although we recognize that this distribution method may not reach all students, we are confident that the diversity in rank and employment status of instructors in FYC gives us the necessary respondent population.
The survey items are consistently measuring the same construct (attitudes towards generative AI in writing), and the survey results are consistent over 930 responses. Additionally, the survey items adequately cover the range of content that is relevant to the construct of interest, and they correlate well with other measures of the same construct. We used Qualtrics Text and Stats IQ analysis along with Excel data analysis and visualization. We further plugged our data into the Claude and Bard generative AI large language models (LLMs) to triangulate data. While the generative AI models gave us accurate overall impressions, Qualtrics and Excel analysis provided the deep dives necessary to draw conclusions. Two researchers on our team (James and Jeanne) ran the data sets and agreed on the results of each.
The study context is replicable across many institutions of higher learning. KSU first-year students represent a diversity that can be scaled and generalized to many other doctoral-granting public institutions in the United States. The survey questions can be added to and contextualized to specific institutions while still mainlining the overall construct of measuring student attitudes. Our research team acknowledges, though, that this survey is a temporally constrained measurement that shows student attitudes towards gen-AI in a specific moment in time. Given the exponential growth of AI and its endless means of integration into all aspects of our students’ writing lives, we will need to continue to distribute the survey each semester to determine how student responses change over time.
The reliability and validity of this survey make it a valuable tool for researchers who want to study students' attitudes toward AI use in writing courses. Versions of this survey can be used to track changes in students' attitudes over time, to compare the attitudes of distinct groups of students, and to identify factors that influence students' attitudes.
Our survey results demonstrated internal consistency in regard to reliability. For example, students answered similarly on two questions that we designed to consistently measure the same attitude/construct. Our results discussed below indicate that the survey items are consistently measuring the construct of attitudes toward AI use in writing courses, and that the survey results are consistent within a semester in the fall of 2023, with mass knowledge of gen-AI being just a year old.
Comparison with Students in Introduction to Technical Writing
In our goal towards learning more about student attitudes toward AI use in writing courses, we wondered if there would be any notable differences between the students enrolled in FYC and our introductory course in technical writing, TCOM 2010. The Department of Technical Communication and Interactive Design (TCID) is a standalone department. It is not housed within English or under the umbrella of “writing studies”. TCID is the home of a BS in Technical Communication; it is this degree program that delivers the service class, TCOM 2010: Technical Writing, to students.
Kennesaw State University is home to the colleges of Engineering and Computer Science/Software Engineering as well as the school of Architecture and Construction Management. While Architecture does not require a technical writing class, Engineering, Computer Science/Software Engineering, and Construction Management include TCOM 2010 in their required curricula. In some cases, this inclusion serves to meet the requirements of an accrediting body (i.e., ABET); in other cases, the inclusion of TCOM 2010 serves to provide students with the necessary communication competencies for a STEM workplace.
We surmised that our population of primarily STEM students could provide responses that differ from the general FYC population. STEM students may view a new technological advance, such as a tool that assists with writing, as very valuable. Additionally, their STEM mindset could reveal dimensions in attitudes that may not appear in the FYC population. As the survey was based on gathering data on student attitudes on gen-AI, we determined we had an invaluable opportunity for comparison.
While FYC students represent the overall university population well, TCOM 2010 classes are primarily taken by students in Computer Science, Industrial Systems Engineering, Software Engineering, Construction Management, and Information Technology. TCOM 2010 classes consist, on average, of 65% male and 35% female; the content of the course centers on the genres of technical communication in addition to audience analysis, and document presentation.
Using the same survey developed to assess the attitudes of FYC students, the Chair (Laura) and the Technical Communication Program Coordinator reached out to faculty teaching TCOM 2010 and, following the lead of FYC, encouraged them to send the link to their students via email, LMS post, or classroom time. As of October 15, 2023, we received 81 responses from the 712 students enrolled in TCOM 2010. This matches the 11% response rate from the FYC student population. The analytical methods for FYC were applied to the TCOM 2010 data set
As we describe and explain preliminary results, we understand the shifting nature of our topic as well as the fluidity of conversations surrounding artificial intelligence. We address our findings within a “moment in time,” the specific temporal space in which we are writing up the results as we have them. We do this with a deep understanding that at the moment we write our research study, there are still responses to the surveys coming in and there are still thousands of conversations yet to be had.