Can ChatGPT pass college assignments? We tested it out, with help from Wisconsin professors

In the era of artificial intelligence, cheating is only getting easier for students.

Some instructors say they can easily tell when students turn in AI-generated work. Others find it far trickier and will turn to online AI detectors for confirmation when their suspicions are raised. Educators everywhere are trying to create AI-proof assignments.

"The more conventional prompt and writing you ask students to do, the more likely they can bring in a machine to do that work," said Chuck Lewis, a Beloit College English instructor and director of the writing program.

The Milwaukee Journal Sentinel tested how well AI can complete college-level work — and whether instructors can detect it.

Where did idea come from?

A Harvard student last year asked seven professors and teaching assistants to grade essays written in response to a class assignment. To minimize response bias, the student told instructors the essays might have been written by herself or by AI, but in reality, all of the work was done by GPT-4, a version of the chatbot from OpenAI.

The AI-generated assignments received mostly A’s and B’s, along with one Pass.

"Not only can GPT-4 pass a typical social science and humanities-focused freshman year at Harvard, but it can get pretty good grades," the student wrote in an essay published by the Chronicle of Higher Education.

How did Journal Sentinel's ChatGPT experiment work?

I followed the same methodology as the Harvard student.

Professors emailed me a smaller assignment they would give their students, not an end-of-the-semester research paper. I told them some of the work would be done honestly and other assignments handled by ChatGPT. In fact, AI did all of the work.

I formulated prompts for ChatGPT from the assignments provided. In most cases, I wrote more tailored prompts to ChatGPT based on what it produced on the first try. Often, the additional requests asked the chatbot to provide more specific examples, expand on its ideas or use a less formal tone.

The experiment was far from scientific. Several professors said they approached grading more skeptically than they would have had it been a student's submission, given the circumstances.

ChatGPT earned 1 A, 2 B's and 2 incompletes.

English assignment at UW-Whitewater

Course: Critical Writing in the Field of English

Assignment: Write a three- to five-page paper examining how a poem among a selection provided draws on a specific concept discussed in class. Include analysis of specific passages in the poem and explore the use of at least five literary terms.

Was this hard for ChatGPT: At first, the chatbot analyzed a completely different poem than the title provided. I submitted the full lines of the correct poem, prompting the chatbot to apologize for the "oversight." Additional prompts providing specific literary terms for the chatbot to incorporate into the essay helped refine the work.

Grade: B+

Comments: The instructor said the paper "fulfills the assignment admirably, and brings an admirable depth of understanding" of the poet's use of the concept. The thesis statement could have been more specific, resulting in a slight deduction.

Political science assignment at Marquette

Course: Introduction to American Politics

Assignment: Write a short paper describing the three faces of power and explaining how each constrains you in your own life.

Was this hard for ChatGPT: No. The chatbot easily put together an essay. A second prompt asking to connect the faces of power concept to my life as a reporter provided more specificity.

Grade: Incomplete

Comments: "Without question, the submission deserves an A," the instructor said. But ChatGPT made one small mistake, which immediately sparked skepticism. While the essay correctly cited the creator of the theory, the reading associated with the assignment was from a different person.

The instructor ran it through two AI detectors, both of which suggested the work was AI-generated. He said he would confront a student who submitted this work.

Library and information studies assignment at UW-Madison

Course: Information Divides and Differences in a Multicultural Society

Assignment: Daily log of media consumption with analysis of tone, evidence, expertise of each source, roughly 350 words

Was this hard for ChatGPT: No. I submitted a second prompt asking for a less formal tone. While the chatbot cited legitimate news outlets, such as the Wisconsin State Journal and New York Times, in the log, the summaries described general topics, not actual news stories.

Grade: 5/5

Comments: The instructor said there were no "egregious red flags" but one sentence stood as sounding like ChatGPT. Overall, he found the discussion of each media source "thoughtful." The log as a whole was "clear and concise." In general, he tends to give students the benefit of the doubt and wouldn't have suspected this log was AI-generated had it been turned in among a stack of others.

Education assignment at Marquette

Course: Child and Adolescent Development and Learning

Assignment: Find five sources for a research paper on an education topic of your choice, cite the sources in APA formatting and summarize each source in four to six sentences. (This is one part of a longer-term research paper.)

Was this hard for ChatGPT: No. It took just one prompt to produce what was submitted.

Grade: 88 (B+)

Comments: The instructor suspected at least some of the summaries were AI-generated because many were written in general terms. However, some of the sources cited appeared to be names of researchers he recognized, making him less suspicious overall.

"It raises an interesting question to me of if students use AI at a broad level (say putting directly in the prompt) or doing a more targeted approach," the instructor said. "I think if students searched for articles and then asked AI for summaries, I might be less likely to have an alarm bell go off in my head."

English assignment at Beloit College

Course: Introductory Literary Studies

Assignment: Write a 1,000-word essay about a selected novel with your own original analysis that engages with a New Yorker review of the novel. Choose from one of three selected topics as a starting point. Include at least three direction quotations from the novel. Give your paper a thoughtful title and underline your thesis statement.

Was this hard for ChatGPT: This was the most challenging of the assignments received and required several prompts to produce. The first essay was only 600 words and included just two direct quotations from the book. It cited a line from a book review that did not actually appear anywhere in the book review. Attempts to expand the essay's length were unsuccessful. An instruction to underline the thesis statement was not completed.

Grade: Incomplete.

Comments: "I didn't need to read more than the first sentence to know this was going to be an AI ride," the instructor said. The writing was "bland," filled with "eager-to-please and generic book blurb-speak." It reminded him of what he tells students: Don't write a thesis nobody can disagree with — there's no argument to be made. The paper also made at least one error in describing part of the book.

The instructor said he would ask the student about their process, share his suspicion, have a discussion and decide on next steps. For now, no grade was assigned.

Contact Kelly Meyerhofer at kmeyerhofer@gannett.com or 414-223-5168. Follow her on X (Twitter) at @KellyMeyerhofer.

This article originally appeared on Milwaukee Journal Sentinel: Can ChatGPT complete college work? Wisconsin professors tested it out