Directory Assistance II

I have occasionally written (examples here and here) about students interpreting assignment and exam prompts in ways that differ from what I intend.

This happened again with the first exam in the undergraduate course that I am teaching this semester. The exam prompt directed students to add to a Twine story. In a class of nineteen students, only one actually wrote text to add to the story. The rest of the students wrote up to three pages that described additions to the story. So here is the prompt for the second exam — changes in bold:

“Play the [link to Twine HTML file in Canvas course shell] game. Write a brief paragraph about one character in the Twine that continues the text of the story and presents the reader with a binary yes/no choice to make about the character. Then write a brief paragraph for each outcome of that choice.  The three paragraphs need to be part of a plot line that reflects one of the following economic development concepts:

[list of concepts students are being tested on]

Write the story, do not describe it.

At the top of your exam, in four sentences or less, 1) identify which of these concepts your plot line demonstrates, and 2) explain how the concept is demonstrated by your plot line.

Your work will be assessed according to the rubric below.”

The second exam is at the end of this week, so I will soon be able to report on whether the revised prompt is more effective.

Collecting Data From Students

As previously discussed, this semester I am attempting to research whether metacognitive exercises improve students’ learning — as measured by exam scores. My class completed the first survey and exam. A few initial impressions about the data and the process of collecting it:

Eighteen students completed the pre-exam survey a total of twenty-seven times. Two students submitted responses three times each. This demonstrates the importance of requiring that students include some kind of identifying information when they complete the survey, so that duplicate submissions can be removed from the data set.

I suspect the survey data are skewed because of above average effect or subject bias. By coding the responses from 1 to 5, with 1 being “never” and 5 being “always,” the highest possible sum score from the ten survey questions is 50. The average for this survey was 40. I doubt students actually engaged in the study strategies referenced in the survey as frequently as they said they did.

The average total score on the exams five multiple choice questions was 7.7 out of 10. Given the small sample and the nature of the data, a statistical analysis that compares these scores against survey responses isn’t meaningful, but I did run a correlation in Excel, which resulted in a very non-impressive r of -0.12.

The exams in this course are extremely low stakes — the first and second exams are worth 25 points each, and the final exam is worth only 40 points, out of more than 1,000 points available from all graded items. That might have affected how diligent students were in studying for the exam.

Given the small size of the class and the usual host of possible confounding variables, I can already guess that I won’t be able to identify a relationship between the metacognition surveys and students’ exam performance. Repeatedly asking students about their study techniques might help them learn more, but I’m not going to be able to demonstrate it.

Failing to succeed

One thing that has been really good about being part of ALPS has been the community around it.

For example, this week’s post is inspired by my former colleague and general force of nature, Maxine David, who pushed out this thread the other day (click to read it all):

Essentially, Maxine’s asking the same question that I think we’ve all asked at some point: what are we trying to achieve in our classes?

As you’ll see from the responses to the thread, I started to sketch out a position, but I’d like to expand on it here some more.

Amanda and Nina have long championed failure in the classroom as a valuable learning experience for students. Their argument – which I also hold to – is that hitting nominal targets is good, but not a complete education: not hitting them encourages students to reflect more on the process of learning (and application) that they’ve undertaken. Think of it as being analogous to playing a game, where not hitting the (rather different) target makes you go back and try again, with the thought of why it didn’t work before in your mind.

This model requires us to acknowledge that learning has multiple targets.

Yes, we want students to know stuff and know how to do stuff (which we can catch with summative assessments), but we also want students to know how to know all this. Becoming a reflexive learner and a critical thinker is a core skill for building capacity to learn throughout the rest of one’s life and it’s a skill that has no easy metric, nor any obvious threshold.

And thresholds were my first thought when I read Maxine’s thread.

When we assess, we typically look for evidence of meeting some threshold: does the student demonstrate that they know enough about X or enough about how to do Y? Those thresholds are present in our grading and those institutional matrices that benchmark us all to common standards.

[Cough]

Maxine rightly points out that we cannot really ever separate out the formative and summative elements of assessment: if we genuinely value the development of reflexive learning, then we absolutely shouldn’t be trying to separate them out in the first place.

But this position is vanishingly rare in academia these days. Yes, I tell my doctoral students that a good viva should see every singly person coming out of the room having learnt something, but even that’s not a given.

Easy as it would be to blame the pressures of QA culture and metrification for all this, we also have to recognise that we often don’t create opportunities within our own classes. Even if we aren’t allowed to make adjustments for support received (as Maxine suggests), we should still be trying to instil a culture of collaboration, reflection and development among our students and between them and us.

In so doing we might start to reclaim some of that learning opportunity that will serve everyone in the class well, wherever they are and whatever they do.

UPDATE:

You might have seen that England is going through some very pointed discussions about racism, following the European football championships. This tweet from one of the national team players exactly captures the point:

Formative Assessment: Abort, Retry, Fail?

Two can play this game

Something of a response to Simon’s June 1 post on transitioning from pedagogical theory to teaching practice: he wrote, in part, “assessment is always formative and should be always linked to the feedback and adaptation process.” In theory, I agree. In practice, while I can lead students to feedback, I am still unable to make them read it.

As I’ve written before, the Canvas LMS has a “student viewed” time stamp feature that shows whether a student looks at my feedback on an assignment — my comments and a tabular rubric with cells that I’ve highlighted — after I have graded it. Generally, though, given the lack of time stamps, many students simply ignore this information. An example, with data: my annual spring semester course on comparative politics. In 2018 and 2019, I taught this course in the physical classroom. In 2020, the latter half of the course was online because of the coronavirus pandemic. In 2021, the course was delivered online for the entire semester. For each iteration, I tallied the number of students who looked at the first three, the third to last, and the second to last reading responses after I graded them. Results are below. N is number of students in the class; not every student in a class completed every assignment. The eyeball columns indicate the how many students viewed an assignment after I had graded it; the eyeball with a slash is the opposite.

While I can understand students not bothering to revisit assignments that they earned full marks on, I don’t understand why students who earn less than full marks frequently ignore information that would allow them to do better in the future. Anyone have an explanation?

Improving Simulation Efficacy With a Scaffolded Final Exam

A follow-up to my post in April about making exams exercises in active learning:

From the very beginning of my teaching career, I’ve emphasized, or at least tried to emphasize, the importance of being able to construct evidence-based arguments. My exams are almost always intended to evaluate students’ proficiency at this task. As I mention in the post linked to above, the final exam for my comparative politics course in Spring 2020 included the stock phrase of:

reference course readings to support your argument.

For the final exam in Spring 2021, I substituted:

support your argument with 1) information from the Gerkhania server on Discord, and 2) cited references to at least two red and two green journal articles listed in the syllabus.

Explicitly requiring the citation of four articles that students were nominally already familiar with from previous assignments resulted in greater use of scholarly evidence in exam essays than had typically occurred in the past. Students sometimes didn’t use these sources in ways that actually supported their arguments, but in these cases I could tell that at least an attempt had been made.

However, to my surprise, not a single student referred to specific player behavior during the simulation. That is not how students read “information from the Gerkhania server on Discord.” Instead, they summarized the simulation’s outcome or, worse, repeated the general background information on Gerkhania that I had provided before the simulation began. So, for 2022, the exam prompt should probably include something like:

support your argument with 1) examples of specific actions made by players during the Gerkhania simulation, and 2) cited references to at least two red and two green journal articles listed in the syllabus.

This is all well and good, because my main purpose for the final exam is summative assessment of learning. But I also want the final exam to help me gauge whether the Gerkhania simulation contributed effectively to this learning. While the first part of my potential exam prompt gets at this question indirectly, I think more is needed. So I have been thinking about “scaffolding” the final exam around the simulation.

I typically run Gerkhania over three sessions. It occurred to me that I could assign something like the following after each session:

Which theoretical perspective best explains players’ behavior in today’s Gerkhania session? Why? Refer to specific player actions and Course Reading X in your response.

These assignments would be short pieces of writing, easy for students to complete and for me to grade. They would allow students to practice for the final exam, and they would function as a contemporaneous reflective through-briefing rather than just a post-hoc debriefing. And I would be able to observe whether students’ ability to construct evidence-based arguments about the simulation improved over time.

Can Final Exams Be Active Learning Exercises?

More on the recent topic of active learning strategies that don’t involve simulations . . . but first some meta-babble on how I understand the concept:

To be considered “active learning,” a task should require learners to apply prior knowledge in a novel way or within an unfamiliar context — what the cognitive scientists refer to as transfer. The activity should, in some fashion, resemble lived or expected experience, because people learn more if they see what they are learning as relevant (a feature referred to as authenticity). The activity should also be organized so that learning occurs efficiently. Finally, the learner should be cognizant of the learning process, which means reflecting on what has been learned and why.

In the social sciences and humanities, exams, especially final exams, are rarely regarded as opportunities for active learning. Perhaps they should be.

Continue reading “Can Final Exams Be Active Learning Exercises?”

Going digital

“This should improve our module evaluations by 0.4…”

Source

The big question of how we forward (not back) in our teaching practice is one that continues to bother me, partly because it’s going to be a major personal challenge for me in the coming years, but also because the variety of discourses about this vary rather more than I’d expect.

As a case in point, I noticed that my VC/President wrote a long blog about this question just the other week. In it, he writes about the possibilities that digital technologies open up and how we need to be receptive and pro-active in making the most of these.

And that’s all fine.

However, what strikes me about the piece is that there’s no mention at any point about pedagogy. Instead it posits a system driven by what the tech can do.

Having gotten to spend some time with him, I know that he does have a genuine and deep interest in teaching in itself, rather than simply as a side-show from research or a money-generating activity (unlike some VCs I’ve encountered), but it’s a bit disappointing to see a senior leader get caught up in the tech.

Tech matters. This past 12 months have demonstrated that all too clearly, but tech is (and can only ever be) a function of pedagogy. More precisely, the fundamentals of good pedagogic practice – clear learning objectives; alignment of content and assessment; responsive design – are just that, fundamentals.

Consider last spring, when you were scrambling around for a means to continue your classes. You probably had an institutional VLE or platform intended specifically for that purpose, plus access to some other tools, either supported by your institution or not.

In the first instance, I’m guessing you took the path of least resistance in setting up ad hoc ways to get content to students and/or having interactions with them.

But then you started to look around at the world of possibilities, just like my VC is suggesting. But in making your new choices, the key driver was likely to have been “what works best for my and my students’ needs” than it was “what amazing thing is possible here”.

In twenty-something years of teaching, I’ve gone from acetates to Zoom, blackboards to Google Docs. But I can think of very few technologies that have fundamentally changed how I teach and only one that changes what I’m trying to achieve with my teaching.

The one change in objectives was the arrival of the digital world and the cornucopia of data that made available. The result was a need to shift from prioritising the acquisition of techniques to find data to stressing ways of managing all-too-much data. And even then, I still find myself telling students how to track down hard-to-find sources.

But otherwise, the bulk of my learning objectives are the same: building substantive knowledge of a topic; acquiring and using skills that make the student into a critical learner; situating all of this within a wider body of understanding.

In short, tech is a means, not an end.

Again, I’ve tried lots of different technological options: some have been great, others alright, a few rubbish. But I could only judge that against the yardstick of my pedagogy and the learning of my students. Great that they can make a whizzy Prezi, but does it actually help them to learn? And I say that as someone who’d love experience more engaging presentations.

So, in the time-honoured cliché of science-fiction, we have to stop wondering what what we can do and start thinking about whether we should do it.

If not, then we risk falling into another cycle of expensive tech acquisition that doesn’t work for our needs, just like we did most of the other times our institutions bought some tech.

Comparing the performance of on-campus and online students in an accidental experiment

This guest post comes from Patrick Bijsmans (Maastricht University) & Arjan H. Schakel (University of Bergen)

Students and staff are experiencing challenging times, but, as Winston Churchill famously said, “never let a good crisis go to waste”. Patrick recently led a new undergraduate course on academic research at Maastricht University (read more about the course here). Due to COVID-19 students could choose whether they preferred online or on-campus teaching, which resulted in 10 online groups and 11 on-campus groups. We were presented with an opportunity to compare the performance of students who took the very same course, but did so either on-campus or online. Our key lesson: particularly focus on online students and their learning.

In exploring  this topic, we build on our previous research on the importance of attendance in problem-based learning, which suggests that students’ attendance may have an effect on students’ achievements independent from students’ characteristics (i.e. teaching and teachers matter, something that has also been suggested by other scholars). We created an anonymised dataset consisting of students’ attendance, the number of intermediate small research and writing tasks that they had handed in, students’ membership of an on-campus or online group, and, of course, their final course grade. The latter consisted of a short research proposal graded Fail, Pass or Excellent.

316 international students took the course, of which 169 (53%) took the course online and 147 (47%) on-campus. 255 submitted a research proposal, of which 75% passed. One of the reasons why students did so well – normal passing rates are about 65% – might be that, given that this was a new course, the example final exam that they were given was one written by the course coordinator. Bolkan and Goodboy suggest that students tend to copy examples, so providing them may therefore not necessarily be a good thing. Yet students had also done well in previous courses, with the cohort seemingly being very motivated to do well despite the circumstances.

But on closer look it’s very telling that 31% of the online students (52 out of 169) did not receive a grade, i.e. they did not submit a research proposal. This was 9.5% for the on-campus students (14 out of 147)[1]. Perhaps this is the result of self-selection, with motivated students having opted for on-campus teaching. Anyhow, it is clear that online teaching impacts on study progress and enhancing participation in examination among online students needs to be prioritised by programme directors and course leaders.

We focus on students that at least attended one meeting (maximum 6) and handed-in at least one assignment (maximum of 7). Out of these 239 students, 109 were online students (46%) and 130 on-campus (54%). Interestingly, on average these 239 students behaved quite similarly across the online and on-campus groups, they attended on average 5 meetings (online: 4.9; on-campus: 5.3) and they handed-in an average of 5 to 6 tasks (online: 5.0; on-campus: 5.9).

We ran a logit model with a simply dummy variable as the dependent variable which taps whether a student passed for the course. As independent variables we included the total number of attended meetings and the total number of tasks that were handed-in. Both variables were interacted with a dummy variable that tracked whether students follow online or offline teaching and we clustered standard errors by 21 tutor groups.

Unfortunately, we could not include control variables such age, gender, nationality and country of pre-education. This would have helped to rule out alternative explanations and to get more insight into what factors drive differences in performance between online and offline students. For example, international students may have been more likely to opt for online teaching and may have been confronted with time-zone differences, language issues, or other problems.

Figure 1: Impact of attending class on the probability to pass

Figure 1 displays the impact of attending class on the probability to pass for the final research proposal. The predicted probabilities are calculated for an average student that handed-in 5 tasks. Our first main finding is that attendance did not matter for online students, but it did for on-campus students. The differences in predicted probabilities for attending 3, 4, 5, or 6 meetings are not statistically significant (at the 95% confidence level) for online students but they are for on-campus students. Students who attended the maximum of six on-campus meetings had a 68% higher probability to pass compared to a student who attended 3 meetings (89% versus 21%) and a 52% higher probability to pass compared to a student who attended 4 meetings (89% versus 37%).

Figure 2. Impact of handing-in tasks on the probability to pass

Figure 2 displays the impact of handing-in tasks on the probability to pass for the final research proposal. The predicted probabilities are calculated for an average student that attended 5 online or on-campus meetings. Our second main finding is that handing-in tasks did not matter for on-campus students, but it did for online students. The differences in predicted probabilities for handing-in 4, 5, 6, or 7 tasks are not statistically significant (at the 95% confidence level) for on-campus students but they are for online students. Students who handed-in the maximum of seven tasks had a 51% higher probability to pass compared to a student who handed in four tasks (69% versus 18%) and a 16% higher probability to pass compared to a student who handed-in five tasks (69% versus 53%).

Note that we do not think that attendance does not matter for online students or that handing-in tasks does not matter for offline students. Our dataset does not include a sufficient number of students to expose these impacts. From our previous research we know that in general we can isolate the impact of various aspects of course design with data from three cohorts (around 900 students). The very fact that we find remarkably clear-cut impacts of attendance among on-campus students and of handing-in tasks for online students for a relatively small number of students (less than 240) reveals that these impacts are so strong that they surface and become statistically significant in such a small dataset as ours.

This is why we feel confident to advise programme directors and course leaders to focus on online students. As Alexandra Mihai also recently wrote, it is worth investing time and energy in enhancing online students participation in final examinations and to offer them many different small assignments to be handed-in during the whole time span of the course. This is not to say that no attention should be given to on-campus students and their participation in meetings but, given limited resources and the amount of gain to be achieved among online students, we think it would be wise to first focus on online students.


[1] The difference of 21% in no grades between online and offline students is statistically significant at the 99%-level (t = 4.78, p < 0.000, N = 314 students).

Collaborative Quiz Experiment

Last week I gave a surprise collaborative quiz to one class, as a test run for possibly using this exercise in my synchronous online courses next semester. The quiz consisted of five multiple-choice questions on basic concepts, deployed in three iterations. First, students took the quiz individually on Canvas, which auto-graded students’ answers but did not reveal which were correct. The individual quiz was worth up to half a percent toward the course grade.

Second, I sent students into team breakout rooms to confer and decide which answers to submit as a group. This version of the quiz was also worth up to half of the course grade. I pasted the quiz into each team’s notes on Google Docs. Because the Canvas quiz tool does not have a “groups” setting, I had already created a Canvas assignment through which each team could submit its answers. Again students did not know which answers were correct — after class I had to read what teams had submitted and manually enter a quiz score for every student who had been present for the breakout room discussions.

Third, after breakout rooms closed, students answered the quiz’s questions yet again in the form of a Zoom poll. After closing the poll and sharing the results, I explained which answers were correct and offered to answer any questions.

Twenty-nine undergraduates are in the course. Three were completely “absent” — they never signed into Zoom during class that day. A fourth student logged out before I announced the group version of the quiz. For the remaining twenty-five students: twelve, or nearly fifty percent, scored higher on the collaborative quiz than on the individual quiz. Great! Three students, all members of the same team, scored lower on the former than on the latter. Ten students’ scores were unchanged.

Finally, the poll, which did not contribute to the course grade: One student left class by disconnecting from Zoom when breakout rooms closed. Of the remaining twenty-four students, nine got the same number of questions correct on the poll and the individual quiz. Ok. Three students did better on the former than they did on the latter. Good. Twelve scored worse on the poll. Terrible! I have no idea why this happened, given the improvement in scores on the collaborative quiz.

Challenges in Using Policy Briefs for Assessment

Today we have a guest post from Simon Lightfoot, Professor of Politics and the Faculty of Social Science’s Pro-Dean for Student Education, University of Leeds. He can be contacted at S [dot] J [dot] Lightfoot [at] leeds [dot] ac [dot] uk.

In recent article I reflected on my use of policy briefs as an assessment task on a final year module Politics of Aid. What surprised me as I was writing the article was how many expected and unexpected challenges arose because of decisions I had made in the design of the module.

I expected there to be some student concern about a relatively novel assessment task introduced in the final year. To counter that I encouraged at first and then made obligatory the submission of a draft. The unexpected challenge was that many students were unsure as to how the deal with drafts. There were always calls for more generic advice despite the fact that each student was able to get tailored individual feedback on their draft.

I thought that students would welcome individual feedback but overlooked the fact that personalised feedback can be specific and very personal. Just as academics bemoan the infamous reviewer 2 (who turns out to have a twitter feed dedicated to their special brand of reviews), students receive feedback in the same personalised way we do. It became clear that we need to ensure that students know how to use feedback if drafts are to be beneficial, and that students need to understand that revision of one’s writing is a central part of the research process.

The drafting/redrafting issue has grown in significance now that the policy brief is 100% of the module assessment. Though intended to reduce the assessment burden on students, the change just raised the stakes—it became a one strike assessment task, which caused some students to feel more pressured. At 1,500 words, the policy brief is very short compared to many other assessment tasks, yet it must demonstrate a high level of research, synthesis, and structure, which requires time. Convincing students to dedicate enough time to their writing proved to be another challenge. As Chagas-Bastos and Burges (2018) found, ‘it is consequently necessary to continuously emphasize the importance of revising and editing, actively encouraging students to deliberately think in terms of drafts’ if they are to produce good policy briefs.