Improving Simulation Efficacy With a Scaffolded Final Exam

A follow-up to my post in April about making exams exercises in active learning:

From the very beginning of my teaching career, I’ve emphasized, or at least tried to emphasize, the importance of being able to construct evidence-based arguments. My exams are almost always intended to evaluate students’ proficiency at this task. As I mention in the post linked to above, the final exam for my comparative politics course in Spring 2020 included the stock phrase of:

reference course readings to support your argument.

For the final exam in Spring 2021, I substituted:

support your argument with 1) information from the Gerkhania server on Discord, and 2) cited references to at least two red and two green journal articles listed in the syllabus.

Explicitly requiring the citation of four articles that students were nominally already familiar with from previous assignments resulted in greater use of scholarly evidence in exam essays than had typically occurred in the past. Students sometimes didn’t use these sources in ways that actually supported their arguments, but in these cases I could tell that at least an attempt had been made.

However, to my surprise, not a single student referred to specific player behavior during the simulation. That is not how students read “information from the Gerkhania server on Discord.” Instead, they summarized the simulation’s outcome or, worse, repeated the general background information on Gerkhania that I had provided before the simulation began. So, for 2022, the exam prompt should probably include something like:

support your argument with 1) examples of specific actions made by players during the Gerkhania simulation, and 2) cited references to at least two red and two green journal articles listed in the syllabus.

This is all well and good, because my main purpose for the final exam is summative assessment of learning. But I also want the final exam to help me gauge whether the Gerkhania simulation contributed effectively to this learning. While the first part of my potential exam prompt gets at this question indirectly, I think more is needed. So I have been thinking about “scaffolding” the final exam around the simulation.

I typically run Gerkhania over three sessions. It occurred to me that I could assign something like the following after each session:

Which theoretical perspective best explains players’ behavior in today’s Gerkhania session? Why? Refer to specific player actions and Course Reading X in your response.

These assignments would be short pieces of writing, easy for students to complete and for me to grade. They would allow students to practice for the final exam, and they would function as a contemporaneous reflective through-briefing rather than just a post-hoc debriefing. And I would be able to observe whether students’ ability to construct evidence-based arguments about the simulation improved over time.

Can Final Exams Be Active Learning Exercises?

More on the recent topic of active learning strategies that don’t involve simulations . . . but first some meta-babble on how I understand the concept:

To be considered “active learning,” a task should require learners to apply prior knowledge in a novel way or within an unfamiliar context — what the cognitive scientists refer to as transfer. The activity should, in some fashion, resemble lived or expected experience, because people learn more if they see what they are learning as relevant (a feature referred to as authenticity). The activity should also be organized so that learning occurs efficiently. Finally, the learner should be cognizant of the learning process, which means reflecting on what has been learned and why.

In the social sciences and humanities, exams, especially final exams, are rarely regarded as opportunities for active learning. Perhaps they should be.

Continue reading “Can Final Exams Be Active Learning Exercises?”

Going digital

“This should improve our module evaluations by 0.4…”

Source

The big question of how we forward (not back) in our teaching practice is one that continues to bother me, partly because it’s going to be a major personal challenge for me in the coming years, but also because the variety of discourses about this vary rather more than I’d expect.

As a case in point, I noticed that my VC/President wrote a long blog about this question just the other week. In it, he writes about the possibilities that digital technologies open up and how we need to be receptive and pro-active in making the most of these.

And that’s all fine.

However, what strikes me about the piece is that there’s no mention at any point about pedagogy. Instead it posits a system driven by what the tech can do.

Having gotten to spend some time with him, I know that he does have a genuine and deep interest in teaching in itself, rather than simply as a side-show from research or a money-generating activity (unlike some VCs I’ve encountered), but it’s a bit disappointing to see a senior leader get caught up in the tech.

Tech matters. This past 12 months have demonstrated that all too clearly, but tech is (and can only ever be) a function of pedagogy. More precisely, the fundamentals of good pedagogic practice – clear learning objectives; alignment of content and assessment; responsive design – are just that, fundamentals.

Consider last spring, when you were scrambling around for a means to continue your classes. You probably had an institutional VLE or platform intended specifically for that purpose, plus access to some other tools, either supported by your institution or not.

In the first instance, I’m guessing you took the path of least resistance in setting up ad hoc ways to get content to students and/or having interactions with them.

But then you started to look around at the world of possibilities, just like my VC is suggesting. But in making your new choices, the key driver was likely to have been “what works best for my and my students’ needs” than it was “what amazing thing is possible here”.

In twenty-something years of teaching, I’ve gone from acetates to Zoom, blackboards to Google Docs. But I can think of very few technologies that have fundamentally changed how I teach and only one that changes what I’m trying to achieve with my teaching.

The one change in objectives was the arrival of the digital world and the cornucopia of data that made available. The result was a need to shift from prioritising the acquisition of techniques to find data to stressing ways of managing all-too-much data. And even then, I still find myself telling students how to track down hard-to-find sources.

But otherwise, the bulk of my learning objectives are the same: building substantive knowledge of a topic; acquiring and using skills that make the student into a critical learner; situating all of this within a wider body of understanding.

In short, tech is a means, not an end.

Again, I’ve tried lots of different technological options: some have been great, others alright, a few rubbish. But I could only judge that against the yardstick of my pedagogy and the learning of my students. Great that they can make a whizzy Prezi, but does it actually help them to learn? And I say that as someone who’d love experience more engaging presentations.

So, in the time-honoured cliché of science-fiction, we have to stop wondering what what we can do and start thinking about whether we should do it.

If not, then we risk falling into another cycle of expensive tech acquisition that doesn’t work for our needs, just like we did most of the other times our institutions bought some tech.

Comparing the performance of on-campus and online students in an accidental experiment

This guest post comes from Patrick Bijsmans (Maastricht University) & Arjan H. Schakel (University of Bergen)

Students and staff are experiencing challenging times, but, as Winston Churchill famously said, “never let a good crisis go to waste”. Patrick recently led a new undergraduate course on academic research at Maastricht University (read more about the course here). Due to COVID-19 students could choose whether they preferred online or on-campus teaching, which resulted in 10 online groups and 11 on-campus groups. We were presented with an opportunity to compare the performance of students who took the very same course, but did so either on-campus or online. Our key lesson: particularly focus on online students and their learning.

In exploring  this topic, we build on our previous research on the importance of attendance in problem-based learning, which suggests that students’ attendance may have an effect on students’ achievements independent from students’ characteristics (i.e. teaching and teachers matter, something that has also been suggested by other scholars). We created an anonymised dataset consisting of students’ attendance, the number of intermediate small research and writing tasks that they had handed in, students’ membership of an on-campus or online group, and, of course, their final course grade. The latter consisted of a short research proposal graded Fail, Pass or Excellent.

316 international students took the course, of which 169 (53%) took the course online and 147 (47%) on-campus. 255 submitted a research proposal, of which 75% passed. One of the reasons why students did so well – normal passing rates are about 65% – might be that, given that this was a new course, the example final exam that they were given was one written by the course coordinator. Bolkan and Goodboy suggest that students tend to copy examples, so providing them may therefore not necessarily be a good thing. Yet students had also done well in previous courses, with the cohort seemingly being very motivated to do well despite the circumstances.

But on closer look it’s very telling that 31% of the online students (52 out of 169) did not receive a grade, i.e. they did not submit a research proposal. This was 9.5% for the on-campus students (14 out of 147)[1]. Perhaps this is the result of self-selection, with motivated students having opted for on-campus teaching. Anyhow, it is clear that online teaching impacts on study progress and enhancing participation in examination among online students needs to be prioritised by programme directors and course leaders.

We focus on students that at least attended one meeting (maximum 6) and handed-in at least one assignment (maximum of 7). Out of these 239 students, 109 were online students (46%) and 130 on-campus (54%). Interestingly, on average these 239 students behaved quite similarly across the online and on-campus groups, they attended on average 5 meetings (online: 4.9; on-campus: 5.3) and they handed-in an average of 5 to 6 tasks (online: 5.0; on-campus: 5.9).

We ran a logit model with a simply dummy variable as the dependent variable which taps whether a student passed for the course. As independent variables we included the total number of attended meetings and the total number of tasks that were handed-in. Both variables were interacted with a dummy variable that tracked whether students follow online or offline teaching and we clustered standard errors by 21 tutor groups.

Unfortunately, we could not include control variables such age, gender, nationality and country of pre-education. This would have helped to rule out alternative explanations and to get more insight into what factors drive differences in performance between online and offline students. For example, international students may have been more likely to opt for online teaching and may have been confronted with time-zone differences, language issues, or other problems.

Figure 1: Impact of attending class on the probability to pass

Figure 1 displays the impact of attending class on the probability to pass for the final research proposal. The predicted probabilities are calculated for an average student that handed-in 5 tasks. Our first main finding is that attendance did not matter for online students, but it did for on-campus students. The differences in predicted probabilities for attending 3, 4, 5, or 6 meetings are not statistically significant (at the 95% confidence level) for online students but they are for on-campus students. Students who attended the maximum of six on-campus meetings had a 68% higher probability to pass compared to a student who attended 3 meetings (89% versus 21%) and a 52% higher probability to pass compared to a student who attended 4 meetings (89% versus 37%).

Figure 2. Impact of handing-in tasks on the probability to pass

Figure 2 displays the impact of handing-in tasks on the probability to pass for the final research proposal. The predicted probabilities are calculated for an average student that attended 5 online or on-campus meetings. Our second main finding is that handing-in tasks did not matter for on-campus students, but it did for online students. The differences in predicted probabilities for handing-in 4, 5, 6, or 7 tasks are not statistically significant (at the 95% confidence level) for on-campus students but they are for online students. Students who handed-in the maximum of seven tasks had a 51% higher probability to pass compared to a student who handed in four tasks (69% versus 18%) and a 16% higher probability to pass compared to a student who handed-in five tasks (69% versus 53%).

Note that we do not think that attendance does not matter for online students or that handing-in tasks does not matter for offline students. Our dataset does not include a sufficient number of students to expose these impacts. From our previous research we know that in general we can isolate the impact of various aspects of course design with data from three cohorts (around 900 students). The very fact that we find remarkably clear-cut impacts of attendance among on-campus students and of handing-in tasks for online students for a relatively small number of students (less than 240) reveals that these impacts are so strong that they surface and become statistically significant in such a small dataset as ours.

This is why we feel confident to advise programme directors and course leaders to focus on online students. As Alexandra Mihai also recently wrote, it is worth investing time and energy in enhancing online students participation in final examinations and to offer them many different small assignments to be handed-in during the whole time span of the course. This is not to say that no attention should be given to on-campus students and their participation in meetings but, given limited resources and the amount of gain to be achieved among online students, we think it would be wise to first focus on online students.


[1] The difference of 21% in no grades between online and offline students is statistically significant at the 99%-level (t = 4.78, p < 0.000, N = 314 students).

Collaborative Quiz Experiment

Last week I gave a surprise collaborative quiz to one class, as a test run for possibly using this exercise in my synchronous online courses next semester. The quiz consisted of five multiple-choice questions on basic concepts, deployed in three iterations. First, students took the quiz individually on Canvas, which auto-graded students’ answers but did not reveal which were correct. The individual quiz was worth up to half a percent toward the course grade.

Second, I sent students into team breakout rooms to confer and decide which answers to submit as a group. This version of the quiz was also worth up to half of the course grade. I pasted the quiz into each team’s notes on Google Docs. Because the Canvas quiz tool does not have a “groups” setting, I had already created a Canvas assignment through which each team could submit its answers. Again students did not know which answers were correct — after class I had to read what teams had submitted and manually enter a quiz score for every student who had been present for the breakout room discussions.

Third, after breakout rooms closed, students answered the quiz’s questions yet again in the form of a Zoom poll. After closing the poll and sharing the results, I explained which answers were correct and offered to answer any questions.

Twenty-nine undergraduates are in the course. Three were completely “absent” — they never signed into Zoom during class that day. A fourth student logged out before I announced the group version of the quiz. For the remaining twenty-five students: twelve, or nearly fifty percent, scored higher on the collaborative quiz than on the individual quiz. Great! Three students, all members of the same team, scored lower on the former than on the latter. Ten students’ scores were unchanged.

Finally, the poll, which did not contribute to the course grade: One student left class by disconnecting from Zoom when breakout rooms closed. Of the remaining twenty-four students, nine got the same number of questions correct on the poll and the individual quiz. Ok. Three students did better on the former than they did on the latter. Good. Twelve scored worse on the poll. Terrible! I have no idea why this happened, given the improvement in scores on the collaborative quiz.

Challenges in Using Policy Briefs for Assessment

Today we have a guest post from Simon Lightfoot, Professor of Politics and the Faculty of Social Science’s Pro-Dean for Student Education, University of Leeds. He can be contacted at S [dot] J [dot] Lightfoot [at] leeds [dot] ac [dot] uk.

In recent article I reflected on my use of policy briefs as an assessment task on a final year module Politics of Aid. What surprised me as I was writing the article was how many expected and unexpected challenges arose because of decisions I had made in the design of the module.

I expected there to be some student concern about a relatively novel assessment task introduced in the final year. To counter that I encouraged at first and then made obligatory the submission of a draft. The unexpected challenge was that many students were unsure as to how the deal with drafts. There were always calls for more generic advice despite the fact that each student was able to get tailored individual feedback on their draft.

I thought that students would welcome individual feedback but overlooked the fact that personalised feedback can be specific and very personal. Just as academics bemoan the infamous reviewer 2 (who turns out to have a twitter feed dedicated to their special brand of reviews), students receive feedback in the same personalised way we do. It became clear that we need to ensure that students know how to use feedback if drafts are to be beneficial, and that students need to understand that revision of one’s writing is a central part of the research process.

The drafting/redrafting issue has grown in significance now that the policy brief is 100% of the module assessment. Though intended to reduce the assessment burden on students, the change just raised the stakes—it became a one strike assessment task, which caused some students to feel more pressured. At 1,500 words, the policy brief is very short compared to many other assessment tasks, yet it must demonstrate a high level of research, synthesis, and structure, which requires time. Convincing students to dedicate enough time to their writing proved to be another challenge. As Chagas-Bastos and Burges (2018) found, ‘it is consequently necessary to continuously emphasize the importance of revising and editing, actively encouraging students to deliberately think in terms of drafts’ if they are to produce good policy briefs.

Monday Morning Advice

A few items from elsewhere that might be of help for instructors who are new to teaching online:

First, here are five reasons not to use timed exams. In my opinion thse arguments apply whether one is teaching online or face-to-face.

Second, here are some very practical, easy-to-implement techniques for becoming a better writer — suitable for both professors and students.

Last, methods of identifying whether your employer’s financial viability has suddenly gotten much worse — editorials that I wrote in 2017 and 2019.

The Benefits of Rubrics

Today we have a guest post from Sarah E. James (sarah [dot] james [at] g [dot] harvard [dot] edu), Colin Brown (colin [dot] brown [at] northeastern [dot] edu), and George Soroka (soroka [at] fas [dot] harvard [dot] edu.)

At the 2020 APSA Teaching and Learning Conference, we led a workshop on writing rubrics for the political science classroom. Rubrics may not be a cure-all for student writing, but we do believe that college faculty underestimate their benefits for teaching advanced writing skills. They can also be a powerful tool for developing a mindful teaching practice.

The use of rubrics is extremely common at the K-12 level (at least here in the U.S.), and there is considerable evidence to support their effectiveness at teaching beginning writing skills (Heidi Andrade’s work is a good starting point, for example, here, here, and here). There has been less evidence for their effectiveness at the university level, but the few existing studies point in the same general direction as the elementary ed studies:

  • Rubrics help students learn specific skills, and make them aware of their own learning processes.
  • Rubrics make grading more efficient, while simultaneously increasing student perceptions of how fair the grading is.
  • Rubrics help expose aspects of the “hidden curriculum” in our writing expectations and may help the most disadvantaged writers the most.

Our key takeaway for participants in our workshop: rubrics let you deal with subjective measures, but in a transparent way, and without being arbitrary. Generating a good rubric requires you to be clear about what you actually value, and on what you expect students to be able to demonstrate. From the students’ side, this is a clear signal of where you want them to spend most of their time. From the instructor’s side, this is a good way to make sure that you’re following the adage of “if you didn’t teach it, don’t test it.” And when we think of the kind of genre-specific writing skills we demand of students, this sort of clarity can be extremely helpful for students who may “write well” in a general sense but who may have no experience in how things like evidence, counterarguments, citations, and literature reviews work in political science specifically.

Rubrics can only capture so much, and when you use one, you are limited to only a certain number of skills or aspects in your assessment. At our TLC workshop, the most common concern our participants had was, “what happens if we end up having to give a good grade to a bad paper?” This is a (small) risk, but we encouraged our participants to step back for a second and think about the rubric as a mindful teaching exercise. If a paper feels like it should get a lower grade than the rubric suggests, are there skills that should be included explicitly in the rubric? (They can be added next time!) If not, then what’s causing you to think the grade should be lower—and is it really something that should be entering into your assessment?

For those interested in designing their own rubrics, we provided examples for an introductory and an upper-level course, as well as a worksheet to help in setting it up. Our examples are designed to focus much more on discipline-specific skills (using evidence, critical thinking, professional norms) than on the quality of prose itself, and our instinct (tested to limited effect in our JPSE article) is that this is the most productive use of rubrics in the college-level classroom. But the structure of rubrics allows them to be adapted to the instructor’s aims, whatever they are—and they force the instructor to make those aims clear to themselves and to their students.

The Muddiest Point, Updated

Many of you are probably already acquainted with the muddiest point technique — asking students to identify the one aspect of a lesson or assignment that they are the most confused by. Often this is accomplished by distributing index cards for students to write on. This semester I’m using an electronic version in a 200-level honors course on Asia: a survey on our Canvas LMS, completed in the last few minutes of class on days for which some kind of lecture or discussion is scheduled. The survey consists of the question “What are you most curious or confused about from class today?” Students automatically earn one point toward the final grade by answering it.

With a paperless process, I don’t have to try to decipher students’ handwriting. And I have an archive of students’ responses that I don’t have to transport or store.

Far more importantly, the surveys are demonstrating the difference between my knowledge base and that of my students — which I otherwise would be mostly oblivious to.

For example, my mind automatically defaults to thinking in terms of power, authority, and legitimacy whenever I’m confronted with the task of analyzing an authoritarian state. Or I recall concepts like ethnic identity when discussing nationalism. Or I know that geography is political rather than an immutable law of the universe — as demonstrated by the origins of labels like Far East, Middle East, and Near East. This is not the case with the majority of students in the class, given their survey responses so far.

Seen Exams

Everyone’s working with seen papers…

This past semester I got to try out using a seen exam for the first time.

For those of you unfamiliar with it, you publish the exam paper some time ahead of the sitting date (a week, in this case), so students can prepare their responses, which they then write under controlled exam controls (without notes or materials to hand).

The logic of this is that it provides a more meaningful test of students’ abilities, since they since have to revise, plan and produce, but without the added peril of “I can’t find a question I can do” or “I answered the question wrong”.

Having inherited the format from a colleague, I was keen to try it out, especially since last year’s use of an open-book, online exam had worked very well. Indeed, this year’s module was with the same students.

The practicalities are very simple indeed: an email to the class and a posting on the VLE at the appropriate time, plus being available through the week to answer any queries or clarifications.

The day before the exam I emailed everyone again, just to run through any points that had come up and to remind them again that the format meant some things were different from a ‘normal’ exam.

Firstly, my expectations on factual accuracy would be higher, since they’d have had time to prepare.

Secondly, I’d like to see more references to the literature: not direct quotes, but certainly mention of relevant authors.

And most importantly, I’d expect clear organisation and argument in each of their answers.

So?

Having now finished my marking, I’m able to say a bit about how this all played out.

As with the other format, this approach seems to be good for pulling up the tail of students who might otherwise have found things difficult: even the worse-performing student still produced relevant answers with some detail.

Likewise, the almost total absence of factual errors and of very short answers was a pleasant development, suggesting everyone had actually done work for the exam.

So the knowledge front seems to be positive.

Having seen a few students straight after the exam, I’m not sure that they found it any less stressful though: yes, they knew what the questions would be, but they also noted that they were also conscious I would be marked in line with that, so maybe their extra work wouldn’t count for anything.

While we’ve yet to complete all the feedback cycle, I think that anxiety is understandable, but hasn’t played out. Instead, the performance of the class has been strengthened and their capacity in the subject will be that bit more for future modules they take.

In sum, this exam has further convinced me that closed-book, unseen exams aren’t that useful, either in measuring knowledge or managing student stress: unless I have to use them in future, I’m not going to be.