At ISA a couple of weeks back, I facilitated a Teaching Cafe discussion on AI and Chat GPT’s impact in our classes. Thanks to the Innovative Pedagogy Conference Committee generously allocating us space, several colleagues from a variety of different institutions stopped by to share their thoughts and ask questions about the ethics, practical responses, and positive aspects of this technology. I’m going to share a few of these responses in case they aid others in thinking through how AI will affect their teaching, with the caveat that AI is advancing at a rapid rate and many of the strategies we discussed will be outdated very quickly.
I’ve categorized our conversation into three themes: how to mitigate the impact of AI in our classes; ethics and academic honesty; and leveraging AI to teach.
A close colleague recently discovered her inbox filling with spam at a rate of over 1,000 messages every twelve hours. Buried in the avalanche was a single email from a bank congratulating her on a new account that she hadn’t opened. Upon contacting the bank, she discovered that someone had obtained her social security number, opened the account, and linked it to second account under yet a different name.
This is not germane to my learning.
The strategy is known as “email bombing” — flood someone’s email with obvious spam on the expectation that they won’t notice the one message signaling identity theft. In other words, criminals are maximizing extraneous load to decrease learning.
My university returned to in-person undergraduate instruction last year, but I’m focusing on reducing extraneous load for the fall semester as if I were still teaching these courses online. This means dropping some non-essential content so that students get more opportunities to practice applying a minimum amount of knowledge to achieve a specific learning outcome. For example, in the course that includes game design, students will have five rather than four opportunities to evaluate existing games before finishing the construction of their own game. In the course where students will try to predict the future, they will practice the different elements of good forecasting multiple times. More on this in my next post.
I recently listened to this episode of the Hidden Brain podcast, on using audible clickers to train humans how to throw a frisbee and perform surgery. Clickers seem to be very effective in part because they substitute for other, possibly emotion-laden reactions from the trainer.
I wondered how I might use clickers as a teaching tool, and had an email conversation with a psychology colleague who specializes in behavioral training. Here is the gist of the conversation:
Clickers provide immediate positive feedback for a specific, discrete action within a complex chain of behaviors, without the need to interrupt the chain as it unfolds.
Any process that is composed of multiple, discrete behaviors is amenable to clicker training, as long as the process can be observed by the teacher and the clicks can be delivered within a second or two of observing the targeted response. An activity like writing is probably not suitable for clickers, because the writing process can’t easily be separated into a series of precisely-defined behaviors, and it would require that the teacher continuously observe the student perform the writing task from start to finish.
However, the technique could be applied to something like class presentations — with clicks delivered when students complete important components of the presentation, such as using terminology correctly, answering an important question, speaking at an adequate volume, or making eye contact with the audience.
Clicks should initially be delivered each time the targeted behavior is displayed, but then systematically delivered less often as the behavior becomes more frequent. Likewise, they should only be used to indicate support for a desirable behavior rather than to signal disapproval of an undesirable behavior.
Last, teachers should obtain consent or buy-in from students about the process and its goals.
A colleague who was cleaning out his office gave me a copy of Scientific Teaching by Jo Handelsman, Sarah Miller, and Christine Pfund (W.H. Freeman and Co., 2008). Intrigued by the title, I gave it a quick read. The book contains some general information on active learning and presents a template for organizing faculty development workshops on topics like assessment, but it was not the guide to effective teaching that I had expected. The book does not discuss empirically-backed research on how people learn. At all.
Instead, Handelsman, Miller, and Pfund favorably discuss learning styles, a zombie educational concept that refuses to die. They heavily reference Learning Styles and Pedagogy in Post-16 Learning: A Systematic and Critical Review by Frank Coffield, David Moseley, Elaine Hall, and Kathryn Ecclestone (Learning and Skills Research Centre, 2004) as support for their argument. In the process, they fundamentally mischaracterize the report’s findings.
For example, on page 9, they write that Coffield et al. (2004) “identified over 70 unique approaches to learning styles . . [that] range from models that explain learning styles as innate . . . ‘flexibly stable’ or . . . that contribute to learning efficacy.” Coffield et al. (2004) state very clearly that these are claims made by those who advocate for the concept of learning styles, not that evidence exists for those claims. In fact, when Coffield et al. (2004) examined thirteen commonly used learning-style inventories, they found that twelve did not meet one or more basic criteria for internal consistency, test-retest reliability, construct validity, and predictive validity. They conclude that the field of learning styles ‘‘is bedeviled by vested interests because some of the leading developers of learning style instruments have themselves conducted the research into the psychometric properties of their own tests, which they are simultaneously offering for sale in the marketplace . . . After more than 30 years of research, no consensus has been reached about the most effective instrument for measuring learning styles and no agreement about the most appropriate pedagogical interventions” (p. 137).
The lack of evidence for the existence learning styles was also discussed in detail by Harold Pashler, Mark McDaniel, Doug Rohrer, and Robert Bjork in ‘‘Learning Styles: Concepts and Evidence’’ (Psychological Science in the Public Interest 9, 3 [2008]). They note in this article that adjusting teaching techniques to students’ expressed preferences for particular forms of instruction (i.e., learning styles) does not correlate to observable cognitive or skill aptitudes, and that only a handful of published studies citing the existence of learning styles had conducted valid experimental tests. The lack of evidence for learning styles was also discussed in this 2009 interview with the cognitive psychologist Daniel Willingham.
In sum, Scientific Teaching‘s reliance on a concept that was widely discredited both before and soon after its publication renders it misleading and, therefore, useless.
I stumbled across Teach Students How to Learn by Saundra Yancy McGuire (Stylus, 2015). Like The New Science of Learning by Doyle and Zakrajsek, it contains some useful advice. Here is a brief review:
The bad
The book has an excessive amount of personal anecdote — such as conversations with and exam scores of individual students — but no presentation of statistically significant findings on overall changes in students’ performance. The author also favorably discusses learning styles and the Myers-Briggs inventory, neither of which is scientifically supported. A more concise presentation with a greater emphasis on empirical evidence would be more persuasive.
The good
McGuire’s focus is on teaching students about the benefits of metacognition, including a specific method of introducing them to Bloom’s taxonomy (Chapter 4). Why is this effective? In high school, students earn high grades without much effort, so they enter college suffering from illusory superiority and ignorant of the actual learning process. Coaching students on specific study strategies (Chapter 5) will therefore benefit them. One example: as professors, we typically know what shortcuts to employ to efficiently find and retain information contained in a book. Students, in contrast, may not know what an index is or how to use one. McGuire also rightly discusses the role of motivation in student learning (Chapters 7-9), and she points out that there are both student-related and professor-related barriers to motivation. These barriers can be mitigated by the instructor.
A final comment
The underlying assumption of this book is that students want to learn, and if they are equipped with the right tools, college becomes a more valuable and rewarding experience for them and their professors. While I think this is a noble and generally accurate sentiment, I’m seeing an increasing number of U.S. undergraduate students for whom college is simply a credentialing process. For these students, the diploma is the goal, learning is not.
I remember a few conversations over the years — once during a job interview — on whether it’s better to give students a concept first and specific examples second, or to provide examples first and then the concept. Bokan and Goodboy (2020)* studied this question with an interesting experiment.
Focus on the triangles’ colors for now; I’ll explain how it relates to something called democratic peace theory later.
They randomly assigned 275 students to one of two conditions in which the order of information in a narrative instructional text moved either from (a) concrete examples to abstract definitions or from (b) abstract definitions to concrete examples. Students reported their perceived cognitive burden during the experiment. Bokan’s and Goodboy’s underlying hypothesis was that poorly designed instructional materials increase students’ extraneous cognitive burden, leading to working memory overload and decreased learning.
They found that placing concrete examples after abstract definitions in an assigned text resulted in higher scores on tests of information recall, retention, and application, even when controlling for students’ prior familiarity with the subject and grade point average. Students “scored almost a whole letter grade lower for every point they reported facing a higher working memory overload.” The authors concluded that the order in which information is presented matters for students reading instructional materials, perhaps because people have a “natural tendency to look for organizing principles before they move on to study more detailed information.” When specific examples are presented before the larger concepts to which they pertain, people are forced to keep detailed information in their minds while simultaneously attempting to categorize it.
*San Bolkan & Alan K. Goodboy (2020) Instruction, example order, and student learning: reducing extraneous cognitive load by providing structure for elaborated examples, Communication Education, 69:3, 300-316, DOI: 10.1080/03634523.2019.1701196.
Cognitive load theory is one perspective on learning that can be applied to teaching in this unusual time. The theory sees working memory — the part of the mind that temporarily stores and manipulates information — as a constraint on learning, because it can only manage a few pieces of information at once. Placing a load on working memory is like trying to push water through a pipe with a constant diameter; you can shove only so much water through the pipe at any given time at a given pressure. If the water’s pressure exceeds what the pipe is able to withstand, you get a flooded basement.
What is the square root of 1,137?
There are three types of cognitive load: intrinsic, germane, and extraneous. Intrinsic load represents the essential actions that occur when learning information that is specific to a topic or task. The load varies according to the information’s inherent level of difficulty for the learner, which makes it partially a function of prior learning: the more one has practiced using the information, the less the effort that needs to be expended the next time it’s encountered. Tying shoelaces requires our full concentration when we are in kindergarten; as adults we perform the task almost automatically and can attend equally well to other cognitive demands at the same time.
Germane cognitive load consists of the work of converting the information in working memory to permanent knowledge, or what the psychologists refer to as a schema. You can think of germane cognitive load as the physical actions of a bank employee storing your shoebox full of cash in the bank’s vault for you to retrieve when needed at some future time. Germane cognitive load is the mental effort that occurs when something is actually “learned.”
This weekend I caught up with an old friend. He works for a software company, overseeing the sales team.
Recently, he’s been doing some work with occupational psychologists, to get a better handle on the team’s stress levels. He told me about all this over a cuppa, including the SCARF model, which I’d not heard of.
SCARF is a diagnostic framework for identifying sources of stress, where individuals encounter challenges to their Status, Certainty, Autonomy, Relatedness (being part of the group) and Fairness.
Listening to my friend, telling me how this works for his team (status is the big thing, apparently), I was struck by how this works in the educational context.
For example, one of the reasons why assessment is so stressful is that it hits most of these areas: students might feel success brings status with teaching staff, it’s relatively uncertain, it’s out of their control, and it’s not necessarily a fair way to judge achievement. The gain of a shared experience with other students pales next to all this.
Clearly, there are general lessons about student welfare to be picked up from this model, but it’s also useful to consider how it relates to active learning.
In traditional, transmission-centred approaches, life might appear to be relatively stress-free: most of the time you sit then, soaking up material, with the occasional bouts of panic at assessment time.
By contrast, active learning might be more challenging.
The biggest issue is likely to be the increased requirement for autonomy: active learning requires participation and the production of contributions on a rolling basis. This front-loads requirements on students, at a point where they might feel they know relatively little (raising issues of status (you want to look good in front of friends) and relatedness (you don’t want to get marginalised in the group if you fail)).
Similarly, the relative absence of the instructor means students have to self-regulate more than usual, so fairness might become more of a factor than in a situation where fairness gets imposed from above.
And it’s also worth highlighting that the model points to active learning being more stressful for teaching staff too, with lower status, higher uncertainty and a big hit to autonomy: no longer is everyone doing just what you want of them.
Despite this, I think that active learning’s benefits outweigh these costs.
Firstly, precisely because students are brought actively into the process from the start, they have much more time to prepare themselves for any summative assessment, both in terms of having to consider materials and of practising producing ideas. The stress is spread out, rather than concentrated at the back end.
But equally, if stress is managed properly, it also comes with raised engagement. If we are making our active learning spaces safe (as we always should be), then we are offering students both the opportunity and the tools to manage stress better, which not only points them to thinking more about the matter in hand, but also how to deal with other sources of stress in their life.
We’re helping our students to learn about the world and how to engage with it. That means skills matter at least as much as substantive knowledge. And handling stress is one of those skills. Yes, active learning is more stressful for all involved, but the benefits that flow from that are ones that might serve us all well.
Everyone should check out this important study by Deslauriers et al, published recently in the Proceedings of the National Academy of Sciences and currently open access. It outlines an experiment at Harvard that tested direct learning in an introductory physics class compared to indirect reports of learning. The takeaway is that students reported they learned more during the lecture—but performed better on quizzes taken following active learning sessions. This has tremendous implications for how we do active learning research–and shows the dangers of relying on student reports of how they learn.
In the experiment, students attended 11 weeks of the introductory course together, and then in the 12th week were randomly assigned to two groups–one with an instructor giving a compelling lecture, and the other with a instructor running a session using active learning techniques. The instructors were both well versed in active learning approaches and had experience in giving great lectures. Students took a survey afterward reporting on their learning along with a 12 question quiz on the material (created by a different instructor to prevent teaching to the test). In the following session the instructors changed their method, so each set of students experienced both a lecture session and an active learning session. The material in the lecture and active learning sessions was identical, as was the handout. In the lecture, the instructor worked through slides based on the handout and solved problems with students passively observing and filling in the answers, while in the active learning session students worked in small groups to solve the same set of problems with the instructor offering assistance as needed. As the authors say “students in both groups received the exact same information from the handouts and the instructor, and only active engagement with the material was toggled on and off” (2).
Students reported greater frustration with the more disjointed nature of the active learning exercise, and thought they learned better from the flow of the lectures, but the researchers found that students performed better on the quiz instrument on the material in their active learning sessions.
We’ve often noted when reviewing research on active learning techniques that indirect measures of learning–that is, student reports on their learning–are not ideal, but this study shows us one of the dangers of relying on such instruments. Less than stellar support by students can derail efforts to increase active learning in a particular institution. We need to be more cautious, then, in how we examine and evaluate evidence that supports–or opposes–the use of active learning in the classroom.
This is a review of “Enhancing formative assessment as the way of boosting students’ performance and achieving learning outcomes.” Chapter 8 of Early Career Academics’ Reflections on Learning to Teach in Central Europe, by Nikita Minin, Masaryk University.
Nikita Minin of Masaryk University is motivated by a goal we can all appreciate: ensuring that his students achieve the learning outcomes of his course. In his case, the course is a graduate seminar on theories of IR and energy security and the learning outcomes include improving student skills in critical thinking and writing. He noticed that students in his class did not seem to really improve on these skills during the class, and introduced three teaching interventions in an attempt to fix this.
First, Minin provided more intense instruction on the writing assignments at the start of the course, providing a grading rubric and examples of successful student work. Second, he gave students audio rather than written feedback on their papers. Finally, using a sequential assessment system, the instructor gave formative feedback first and grades much later in the course. Minin assessed the impact of these three interventions, comparing course sections with and without them, and concluded that the first two interventions achieved the objective of improving student achievement of the learning outcomes.
The interventions described in the chapter are in line with current thinking regarding in-course assessment. While Minin does not use the language of transparent teaching, his first intervention falls exactly in line with the Transparency in Teaching and Learning Project’s (TILT)approach. Transparency calls on instructors to openly communicate about the purpose of an assignment, the tasks they are to complete, and the criteria for success, and Minin does exactly that in this first intervention. Given the data so far on the TILT project, it is not surprising that Minin saw some success by taking this approach. Likewise, now-ubiquitous learning management systems allow for giving feedback in multiple platforms, including audio and video. For years now, advocates for audio-based feedback claim that this can be a more effective tool than written feedback. Minin’s observations therefore, also fit nicely in line with existing work.
Where the chapter falls short, then, is not in the design of its interventions, but in the claims made based on the available data. The sample sizes are tiny, with just five students receiving the interventions. With final grades used as the primary dependent variable, it is difficult to tease out the independent impact of each of the three changes. Using final grades is also an issue when the experimenter is also the person who assigns grades, as it is more difficult to avoid bias than when more objective or blind items are used. Lang’s (2016) bookSmall Teaching: Everyday Lessons from the Science of Learningtells us that engaging in self-reflection is itself an intervention, and Minin’s use of minute-paper style self-reflections to assess the impact of feedback, while itself an interesting and potentially useful idea, mean that a fourth intervention was used in the course. While I do not doubt Minin’s observations that his interventions had a positive impact, as they are backed by existing research, the evidence in the chapter does not strongly advance our confidence in those findings.
However, I have never been one to dismiss good teaching ideas simply because of a lack of strong evidence from a particular instructor. Minin highlights a crucial concern—that we should never assume that our courses are teaching what we intend them to teach, and that ‘time and effort’ do not necessarily achieve the desired results, even for graduate students. Reflecting on this, seeking out innovative solutions, and then assessing the impact is a process we should all be following, and Minin sets a great example.
