This guest post is by Dr. Ray Mikell, an assistant professor of political science at Jackson State University, a historically black institution and urban research university in Jackson, Mississippi. He has taught courses and written test bank questions in American and comparative politics in connection with several widely used, introductory-level textbooks, and has worked with W.W. Norton & Co. and Pearson.
Editor’s Note (Pigeon, that is): This series adds to the sober discussion we’ve been posting about using AI in the classroom. In my assessment, this series highlights the importance of subject matter expertise. Ray has many years of experience in deploying relevant and accurate classroom exercise scenarios and thus has appropriate judgment in reviewing, evaluating, and editing LLM scenario outputs. Over to you, Ray!
~
Could having individual students think through scenarios from the worlds of politics and government—about everything from international conflict to voter choices, and a whole introductory political science text laundry list of subjects—in an open-ended fashion help ease educator anxiety about cheating via artificial intelligence systems? Being at least a near-optimist, I wondered if such questions could help increase student skills in analytical and critical thinking as well.
A win-win? Maybe I’m naïve?
This is a conversation I started having with myself—and then a large language model (LLM)-powered online chatbot —in recent times for a couple of reasons: The first spark was faculty anxiety about LLMs and artificial intelligence, which many critics saw as plagiarism systems. (I came to share the unease over the past two years, despite some initial ambivalence.) Then, for freelance American Government test bank work I was doing in 2024, an editor asked me to write more scenario or vignette-oriented multiple-choice questions than usual. The rationale? In an era when LLMs can easily produce basic recall questions, coming as higher-order thinking was becoming more valued, it made sense to experiment.
Any political science instructor who has used publisher test banks over the past decade or two has come across these types of questions, in this case, what education researchers call context-dependent items. These typically introduce a brief story or scene before posing a course-relevant query. Some may be augmented by a data graphic, a photo, or video. Others are formatted as an item set, otherwise known as a testlet. In these (common in medical or health care-related education), a vignette is followed by two or more questions, often linked in step-by-step fashion. In effect, these are puzzle questions aimed at getting students to think critically about political situations and issues as they present themselves in our world.
Maddeningly, however, no research has ever shown whether these sorts of questions are demonstrably effective in increasing critical or higher-order thinking. I should know. In the late 2010s, I undertook context-dependent query experiments with a few introductory-level political science classes at Jackson State. Ultimately, I found only modest effects, at best, as far as knowledge retention and increasing critical thinking went (as judged via essays). Immediate online feedback had a somewhat stronger effect, but not a notably strong one.
Although the results suggested that these types of questions only slowly push students toward more analytical reasoning, I kept slipping scenario questions into student introductory-level classes and tinkering with question design. The rise of ChatGPT, however, led me to doubt how I would go about testing student knowledge and comprehension of course materials through such means.
More recently, amid a presidential administration where the federal government was not working in a textbook fashion, I had a change of heart. Maybe context-dependent questions had more worth than I thought? The major difference, it seemed, was that I could no longer see using essays to judge student recall and comprehension of course concepts and facts.
One way to adapt, I thought, would be to lean into pushing for more analytical thinking through open-ended scenario questions or prompts that students could address via essays, out-of-class research, presentations, and audio-visual content. It would not, I thought, be nearly as easy for students to try to skate by with LLM-provided responses to scenario prompts, given that the latter often can involve thinking through complex situations. Such assignments could require students to apply abstract concepts to real-world problems.
It was possible, of course, that LLMs could provide respectable responses to my prompts in the usual AI way, with clear and shiny, if bloodless, prose, thus proving my naivete. But to judge that, I had to jump right into AI-land. More specifically, I had a chat with Claude AI about the worth of using scenario questions to get around or above cheating and plagiarism. Although it, like all LLMs, aims to please users—which can seem seductively refreshing after reading peer reviews—the LLMs answers were nonetheless intriguing.
Next in this series: How Claude AI answered the prompts (preview #1: too many bullet points, long-winded, but would have kept students from having to worry about more basic concepts or facts about American government), what it had to say in conversation about (preview #2: it suggested incorporating data searches), and the questions about boosting critical thinking that its answers and information pose.
For me, the key sentence in this post is: “I could no longer see using essays to judge student recall and comprehension of course concepts and facts.”
Because of AI, I can’t assign take-home analytical essays anymore. My wife, who teaches English literature, can’t either. As for in-class writing exercises and essay exams, many students write in illegible scrawls, which I can’t decipher, while others have disability accommodations that allow them to use keyboards. So in the end, the rationale for in-class writing becomes moot.