Assessing large groups

Among the many upsides of working with relatively small groups of students for most of my career has been that I’ve not been driven primary by volume management concerns. I could look across at my colleagues in Law or Business and laugh at the thought of having to juggle several hundreds of exam scripts at once.

(One time, a Business lecturer proudly told us how he’d marked 400 exams in three days, only for it to become clear it was a multiple-choice paper, with answers scanned in, which raised questions about why it had taken so long.)

But in the spirit of cooperation, a recent tweet about the need to treat assessment as an integral part of our teaching activity prompted this response from a Legal colleague:

This is an issue for many of us at some point: the big compulsory course where innovation feels like a luxury.

So what do to?

Sylvia’s dilemma is three-fold: assessment needs to a) serve learning objectives, b) minimise opportunities for cheating, and c) be practical to turn around with reasonable speed. We’ve not had the chance to speak about her specific situation, so what follows should be read more generically.

My personal view is that we always have to place learning objectives first in assessment: do we test for the things that we consider it essential that the students should have learnt?

In any course or module that covers a variety of elements: substantive knowledge; research skills; presentational and interpersonal skills; and more general aspects of critical thinking and building confidence. That breadth is important, because it underlines that ‘knowing facts’ isn’t the be-all and end-all here: even for us in academia, we probably make as much use of the skills and competences we gained from our study as we do the knowledge (and we’re at the high end of a spectrum of knowledge use).

Sylvia mentions vivas as a nominally ideal form of assessment, possibly because it’s interactive and personal and offers lots of opportunities to test how far a student can work with what they know. Having sat through vivas for a course of 100 students, I could point up some issues, but the analysis still holds: here’s something that better serves the learning objectives.

So are there other ways to get that same benefit without the big time implications of a viva system?

Two strategies suggest themselves, if we’re treating final written exams as an unsatisfactory option: different formats and collaborative working.

Asking students to produce posters, infographics or podcasts not only opens up different ways of presenting material, but also requires considerable distillation of substantive knowledge into key points, which in turn stimulates more critical engagement. Yes, students will be unfamiliar with the practicalities, but this can be covered with some preparatory sessions, and it develops presentational skills that might be otherwise neglected.

If you want to stick to text, then asking for shorter written pieces – policy briefs, submission to a court – can also keep the focus on distillation, plus give experience in formats they might encounter in their work (unlike a long-form essay).

And all of these options could be used with collaborative approaches too. Learning to work together is a valuable skill [he writes from his shed], so why not test for that? Group projects can be marked for the group as a whole, plus with individual marking for short reflective pieces on what each person contributed and got from it.

Of course, free-riding is an issue, and some disciplines might encounter accreditation barriers on collaborative assessment, but the problems need not be any greater than for final exams.

The right answer will vary from case to case: your capacities; the nature of your course; your institution’s attitude; the willingness of your students to buy into it. But these discussions are still worth having. Just because things have ‘always been like this’, doesn’t mean they should continue like this, especially if it’s not working for you or your students.

If you have more ideas on this, or what to chat about your assessment, drop me a line and we’ll talk.

The Bandit Game

In an attempt to rectify the failure of my previous classroom game on ethnic heterogeneity, democracy and dictatorship, I created another game that included a loss aversion component. I intended the game to demonstrate the concepts found in Mancur Olson’s 1993 article, “Dictatorship, Democracy, and Development” (The American Political Science Review 87, 3). Here are the rules for game’s initial version:

  • Each person gets a playing card and 4 chips.
  • The class is divided into small groups.
  • The person with the highest card value in each group is a bandit.
  • The game has five rounds.
  • Each group’s bandit confiscates 1, 2, 3, or 4 chips each round from every other group member. This decision is made by the bandit. The bandit has to confiscate at least 1 chip from each group member each round, assuming the group member has a chip.
  • After round 1, 2, 3, and 4, each non-bandit gets 1 additional chip if they have ended the round with > 0 chips.
  • The person in each group with the most chips after round 5 earns points equivalent to the number of chips in their possession.

Version 2 of the game has the same rules as Version 1, plus:

  • A bandit can switch to a different group after each of rounds 1-4. The bandit with a higher value card turns another group’s bandit into an ordinary person.
  • The new bandit takes the eliminated bandit’s chips and can keep them or distribute some or all of them in any manner to members of their new group.

Version 3 has the same rules as Versions 1 and 2, plus:

  • Members of a group can eliminate a bandit if (a) they have card suits different from the bandit’s suit, and (b) the combined value of their cards exceeds the value of the bandit’s card. If a bandit is eliminated, the bandit’s chips are distributed equally among the challengers.
  • A bandit can retain control if (a) group members with cards of the same suit as the bandit’s decide to ally with the bandit and (b) the combined value of cards of this suit exceeds that of the bandit’s challengers.

Before play started, I stacked the deck with cards from only three suits because of the small class size — thirteen students are registered for the course, but only eleven showed up. I divided these eleven students into three groups.

For all versions of the game, all bandits confiscated the same number of chips from their group’s members in each round, even though the rules did not specify that they had to do this. In Version 1, one bandit confiscated all the chips from every group member in one round, which ended that group’s game play for the remaining rounds — demonstrating that it’s better for a stationary bandit to extract only a portion of wealth from the populace at any given time. During Version 2, no bandit changed groups, and in Version 3, no one tried to eliminate a bandit.

This game worked better than the last one, but it still needs a much larger number of participants for it to function as intended.

Serendipity in Research Methods

Sometimes it is easier to demonstrate real-world relevance than others.

Last week students in my research methods course read Charles Wheelan, Naked Statistics, Ch. 12, and Ashley A. Smith, “Students Taking More Credit Courses and Introductory Math Faring Well,” Inside Higher Ed, 7 December 2018.

They then had to answer this question: What mistakes are Nevada officials making with data about community college students?

As written, the Inside Higher Ed story describes people who should know better falling victim to omitted variable bias and confusing correlation with causation. Although I might be making similar mistakes in evaluating in-class discussion about the assignment, I think that students found it more interesting than most because the assignment was about other students.

Soon afterward, two similar items came across my radar:

Students prefer mixing and matching online with on-campus courses.

Common premises about college students are wrong.

I shared these with my students, as additional examples of analyzing (or not) data about their peers.

Call for papers: EuroTLC, Bratislava

This June, the 5th Conference on Teaching and Learning Politics, International Relations and European Studies (EuroTLC) will be held in Bratislava, Slovakia.

As with previous EuroTLCs, the aim is to have an open and flexible set of discussions and activities for a wide range of colleagues, so it’s a great opportunity to share your practice and to learn from others. There’s also a ‘garden party’, so even more reason to attend.

This time around, the event is being co-organised by the ECPR, Comenius University and the IMPACT project. Registration is free, as ever, but you will need to have an affiliation with an ECPR member institution.

Deadline for proposals is 7 April, so get your ideas in now!

Study Abroad As Active Learning

Alternative title for this post: What I Did Over Spring Break.

Photo credit:
Chad Raymond

One of the benefits of having a joint appointment in an interdisciplinary department is being able to participate in field research initiatives. Two weeks ago, I was in Belize as a co-leader for a tropical biology course examining the effects of agricultural development on biodiversity. This was a great opportunity to observe and teach about intersections between economic development, environmental sustainability, and public policy — while getting one’s hands dirty among the scorpions, bats, snakes, and octopi.

A few random observations:

Undergraduate political science programs with curricula that exclude experiential learning in favor of scholasticism are really doing their students a disservice. Philosophical treatises are no longer the sole repository of knowledge.

American provincialism did manifest itself, but only occasionally and, in my opinion, innocuously. For example, as this blog’s non-U.S. audience probably already knows, my people are generally at a loss in non-English language environments. But despite English being the official language of Belize, it is English with a Belizean accent. And in everyday conversation among locals, Belizean Creole is used — often mixed with Spanish. Occasionally I asked students, “Did you understand that?” and their answer was “no.” I would then point out that American English is not the only form of English, and that they needed to train themselves for that reality. The few among us competent in Spanish definitely had an advantage when we encountered people who did not speak any version of English. For readers in the U.S., does your political science department require proficiency in a second language? It should. Politics are global.

I was impressed by the students’ initiative and willingness to try to independently solve problems on the fly while doing their research. They were also relatively unperturbed by the conditions. Maybe this is due to self-selection for this specific study abroad program, but whatever the reason, it made my job much easier.

In-country logistics ran like clockwork, mainly due to the talented local program organizer. It was only when returning to the USA, a Third World country, that we ran into problems — a flight delay, a mad scramble through Miami airport, a missed connecting flight, an interminable rebooking process, not all of us finding seats on the last flight of the day, and a portion of our group forced to overnight in an airport hotel. But everyone did return home in the end. Lesson to students: be prepared in case things don’t go according to plan.

The interrelation between attendance and writing assignment in a PBL course

This guest post comes from Patrick Bijsmans (Maastricht University) and Arjan Schakel (University of Bergen)

In one of his recent contributions to this blog, Chad asks why students should attend class. In his experience

[C]lass attendance and academic performance are positively correlated for the undergraduate population that I teach. But I can’t say that the former causes the latter given all of the confounding variables.

The question whether attendance matters often pops up, reflected in blog posts, such as those by Chad and by Patrick’s colleague Merijn Chamon, and in recent research articles on the appropriateness of mandatory attendance and on student drop-out. In our own research we present strong evidence that attendance in a Problem-Based Learning (PBL) environment matters, also for the best students, and that attending or not attending class also has an influence on whether international classroom exchanges benefit student learning.

Last year we reported on an accidental experiment in one of Patrick’s courses that allowed us to compare the impact of attendance and the submissions of tasks in online and on-campus groups in Maastricht University’s Bachelor in European Studies. We observed that that attendance appeared to matter more for the on-campus students, whereas handing in tasks was important for the online students.

This year the same course was fully taught on-campus again, although students were allowed to join online when they displayed symptoms of or had tested positive for Covid-19 (this ad-hoc online participation was, unfortunately, not tracked). We did the same research again and there are some notable conclusions to be drawn.

In the first-year BA course that we looked at, students learn how to write a research proposal (see here). The course is set up as a PBL course, so it does not come as a big surprise that attendance once again significantly impacted students’ chances of passing the course.

Continue reading “The interrelation between attendance and writing assignment in a PBL course”

Learning about teaching

Blossoming

Last night found me at our kids’ school, for a talk on revising. Aside from being a reminder of how quickly people decide that facemasks aren’t prudent any more, it brought home some lessons about the way we construct teaching for others.

The talk was primarily a run-through of what will be happening after Easter with exams, plus subject-specific sections on useful resources and good revision practice. Its content was much as you imagine, and as familiar to me (as a teacher) as it was to my daughter (who’s now on her third time of hearing it all in as many weeks).

So what’s worth mentioning here, on a site devoted to university education, where we don’t (usually) draw parents into it all?

The teachers here have clearly had some training on revision, including some useful models of ‘how to revise’, which they brought to the table. But what was missing (for me at least) was an unpacking of how revision fits into the broader process of learning.

Back at the start of the year, we got a welcome talk about what the next cycle for our daughter would be (the two years up to her first major external exams). In that was lots of stuff, but not so much about how keeping materials and notes would be a key part of ‘revising’, in the sense that they discussed last night. Revision implies vision a first time, and all the revision techniques set out in the current talk require a baseload of substantive knowledge and understanding to be able to produce the materials for effective assessment performances.

Put the other way around, if you’d not done the work until now, having six weeks until the tests to revise as the school would like you to is not a viable proposition.

And this is where this all matters for you (and me). Assessment (and by extension, revision) is too often treated as a disparate element of the educational experience; something tagged on the end, just because we have to.

Instead, assessment is an integral part of learning and should be handled as such, a logical extension to what happens in a class and in a student’s broader package of work through a programme.

This disconnect was evident in a couple of other places too, last night.

One of the teachers asked that students didn’t come to them, asking for ‘help with something vague’, but rather with a precise and focused query: ‘I have tried to do this past paper question on topic X and I can’t seem to make sense of it, despite several tries’, seemed to be the preferred line.

Now, as a teacher, I appreciate that more precision means more scope to get into the nuts and bolts with a student, but I also appreciate that the bigger problem is students not coming to ask for help at all. If I were a student who was struggling, being told I now needed to come with a precise inquiry strikes me as more daunting.

Here the issue is one of assumption-making about student engagement and buy-in to the programme of study. Even the most wonderful teaching set-up does not guarantee that engagement and we always have to be alert to those students that haven’t found their place within it.

That’s best treated not as the fault of the student, or the teacher, but of the specific instance. In a university setting we have more discretion to change and adapt that instance to accommodate individuals on a different path, but in a much more prescriptive system – such as that found in schools – the need to nudge/shove everyone into the same track is much more considerable.

The key take home for me from all of this it that we need to be thoughtful about how we communicate with our students. That means not simply setting out what to do, but rather explaining what we’re trying to achieve (in the broad and narrow senses): it doesn’t stop us from recommending techniques to follow, but it does then require us to explain why these might work.

Since I don’t want to paint our school in a completely bad light, they did do this last night when talking about planning revision. As was explained, prioritising topics is a key first step in making a revision timetable: the focus should be on what’s less comfortable or familiar, because that’s where the biggest gains can be, rather than sticking to the stuff you know.

Of course, sometimes even the stuff you know turns out to be not as simple as you might think.

When a Game Fails

An inadvertent update to a 2015 post on the perils of small classes:

I recently ran a game in two classes that I had hoped would demonstrate the effects of ethnic heterogeneity in dictatorships and democracies. The basic mechanics of the game:

The class is split into groups. Each person gets a playing card. Card suit represents ethnicity, though I didn’t tell students this. A card’s numeric value equates to the power level of the person holding it. If someone in a group has a face card, then the group is a dictatorship. The person in the group with the highest value face card is the dictator, who makes all decisions. If no one in the group has a face card, then the group is a democracy, with decisions made by majority vote. The numeric values of the cards don’t matter.

The game is played in multiple rounds, with a greater number of points at stake in each round — I used five rounds, worth 3, 5, 7, 10, and 15 points, respectively. These points count toward the final course grade. In every round, each group allocates its points to its members according to the rules above. If anyone in a group is dissatisfied with how the points were distributed, the person can recruit a cluster of allies who have cards of the same suit to challenge the distribution. In a dictatorship, the challenge succeeds if the cluster’s combined power level exceeds that formed by the dictator’s allies. In a democracy, the challenge succeeds if the cluster’s total power level exceeds that of the rest of the group. When there is a successful challenge, the group has to distribute its points in a different way. Each round had a time limit of just a few minutes, and if a group failed to successfully allocate its points before a round ended, the group’s points for that round disappeared.

Continue reading “When a Game Fails”

Academics as part of a community

Striking, but probably not enough

The invasion of Ukraine raises many questions, both academic and practical: certainly it has the feel of an event that will shape a lot of lives for a very long time.

Part of that has been the question of how to respond, as an individual and as part of a community.

I’ve had to deal with this a bit more than many, as a function of chairing UACES, the UK’s European Studies association.

We have a few Ukrainian (and Russian) members, plus many more with personal or familial ties, reflecting the entangled nature of European society.

Like many other academic bodies, we issued a statement that condemned the invasion and called on members to help where they could. However, we also didn’t want to just leave things there.

As a result, we’ve carried on working to link up with bodies that can provide direct support, such as CARA and Scholars at Risk, to try maximise the connections that exist, while also opening ourselves up to help people more individually, as best we can.

How much help that ends up providing remains to be seen, but already at this stage it has underlined for me the importance of not falling into a performative trap of simply declaring “X is Bad” and the moving on.

If you have examples of how you see ways we could be doing more, then I’m really keen to hear them. Each of us might not be able to do much to address or remedy the situation, but equally it behoves us all to try where we can.

Creating Wicked Students 2

As promised last week, here is an example of a wicked problem I’ve given to my comparative politics class.

  • You are an employee of the The Scowcroft Group.
  • Volkswagen wants to expand into a new African market.
  • Setting up production facilities and distribution channels will take three years.
  • Which sub-Saharan African country should Volkswagen choose to expand into?
  • Your task is to compare risk to political stability for two sub-Saharan African nation-states, and choose the one with the least risk.
  • Use ≥ 1 quantitative and ≥ 1 non-quantitative indicator.
  • Present your recommendation to Volkswagen’s CEO and board of directors.
  • You have 15 minutes to create a 3 minute presentation.

I show the instructions, small teams of students work on the problem, and each team presents its solution. I grade the presentations using this rubric: