Using Oral Exams in Political Science

Happy New Year to all.  I hope our collective breaks have been enjoyable and that we are reinvigorated for the start of spring term!  I know I could use a couple of extra weeks, but instead I’m finishing up a paper and getting ready for SPSA next week.

I’m a fan of oral examinations in our field.  One of the challenges students face in a written exam is interpreting exactly what we mean in the questions that we ask.  Language that, to us, appears clear and obvious, to them is vague and confusing.  This is even more the case for multiple choice questions, when there is not only a stem to interpret, but four or five possible choices.  No matter how clear we are, students–even, perhaps especially, those who are very prepared for the exam–can overthink the options and choose the wrong answer.  Another problem with written exams is that its not exactly a skill set that students need after graduation.  I can’t imagine many situations where, outside of formal education and standardized entrance/exit exams, they will be asked to prove their knowledge on the basis of prewritten questions that they do not know ahead of time.  I doubt many of us actually grade students on the sentence structure and grammar, so as an exercise in writing its not particularly useful either.

Oral exams, on the other hand, can accomplish the same goals as written exams–testing students’ knowledge and understanding of the material–but have some additional benefits.  As an interaction between professor and student, they can help us dig through the BS that fills exam answers and figure out whether the student really knows what they are talking about.  It also can help us clarify the question or ask it in a different way to help a struggling or confused student better understand what is being asked.  Finally, its pretty decent training for the post-school world, where you may have to defend a political argument before peers or supervisors in the course of a conversation.

We use an oral exam as the capstone project for our majors, called the Overview, so I instituted a mini-overview in my required methods class.  The Overview requires students to sit a 45 minute oral exam with two faculty members on a scholarly book, and face questions about the authors argument, values, and methodology and are asked to synthesize those with ideas and issues confronted in their coursework.  The mini-overview is a miniature version of that.  It acts as the midterm in my methods class, and students read a book chapter or article that I assign (the same one for each student) and sit a 15 minute oral exam with me.  In the course of the mini-exam, we focus almost entirely on methodology and analyzing the argument and I have students break down the variables, measures, methods, sources, etc.  Students also turn in an annotated abstract of 300 words for the piece, with the annotations noting the X, Y, theory, methods, and findings.

I only had 25 students in this class, so it is perhaps easier for me to do individual oral exams than it would be if I had 100 students.  Most of the exams were held during what would have been class time, so it did not take up too much of my time, and grading occurred immediately. If you have larger classes, you could consider training TAs to give these exams, as we trust them with grading anyway.

In general students performed better on this oral midterm than my previous group of students had on their written exam, and this was at least partly due to my ability to probe their responses for understanding, and encourage them to make vague responses concrete.  My first crop of students that underwent mini-overview take their capstone Overview this spring, so I will report back on its longer-term impact then.

What students want

It’s the final teaching week of the semester here at Surrey, so it’s wrapping-up time.  As well as trying to get students to connect their class-based work with events in the wider world, I was also interested in using the final session with them to get some constructive feedback on changes for the next time I teach my module on negotiation.

I’ve talked about the A-B-C feedback system I use before, so I’ll concentrate on what they’ve said.  Essentially, students are asked to suggest one thing to abandon, one to begin and one to continue.

If we remove what might be politely termed ‘outliers’ (e.g. classes in the pub, no assessment at all), then it is very interesting to see what came out.  While each of the individual games I ran had at least one student wanting to abandon it, there was a big group who liked the range provided, especially since it allowed them to practise a range of skills.  Similarly, opinion appeared to be split on the use of a reflective portfolio for the assessment, with some students wanting more structure to it, and others liking the flexibility.

In short, it shows that Lincoln’s adage about pleasing people is just as true in this context as any other and that even in a very student-led learning environment, it is essential for the teacher to provide direction and guidance.

Having said this, the session did bring forward one very good idea that had previously only nestled, half-formed, in the back of my mind.  This is simply to link together all the games into a coherent story (or arc, as I think we have to refer to it now in the age of the box-set), so students can develop into roles over time.  This would neatly solve the disjuncture between games, improve the capacity for preparation, bring out issues of iteration and, as such, was enthusiastically received by the class when we talked about it.  Of course, it also implies another complete re-write of my materials, but that might not be a bad thing.

Design your own Final

A fun way to get the students thinking about the final exam is to give them time on the last day to help write it.  The last day of class is a great time to review and reflect on what the students have learned throughout the semester, but they may have trouble relating that reflective period to the actual final exam.  Having them design the final (or parts of it) forces them to think more systematically about the exam as questions and answers, and also makes them more likely to think of the exam as ‘fair’ since they helped create it. If you aren’t having a final, you can adapt the exercise to help them reflect and synthesize.

The amount of direction I give the students depends on the class.  Usually I have students pair off and give them 5-10 minutes for the exercise.  First, i review what a ‘good’ exam question, whether short answer or essay, looks like.  They usually like this, not only because it helps them write their own questions, but also because it gives them insight into how I create their exams.  Then I assign each group a chapter or theme, and have them write 3-5 questions on their assigned area, but also to write 2-3 questions that synthesize the broad themes of the course or draw on multiple chapters. I promise that anywhere from 1 question to the entire exam will be made of of their questions, depending on the quality of the questions.  Sometimes as an added bonus I throw in the promise of extra credit on the exam for anyone who gets a question on the exam.  I collect the questions, type them up, and post them online as their study guide.

The questions often require a bit of alteration on my part, and I also tend to throw in a few of my own as well.  I add in a section on the format of the exam and some preparation advice, and that’s about it.  It gives students the study guide they want with minimal work for me, gives them some ownership over the exam, starts their minds on the reflective path I want them to take before the final, and sometimes saves me the work of writing exam questions!

The Many Rewards of Teaching

First, my apologies for my absence from ALPS.  A month of trips, administrative deadlines, illness, and conference preparation took their toll.  But fear not! I’m back and ready to talk teaching once again.

I was going to comment on a conversation I had with two groups of students regarding when they do and do not read their assigned texts; I also want to chat about the simulation Statecraft, using games like Diplomacy in IR, a conversation about teaching methods from ISA-MW, and course-long class projects.  So many blog posts, all of which justify my continued avoidance of grading papers!

But all of those ideas have to take a backseat to something wonderful that happened yesterday, just as I was questioning whether the onslaught of work was worth it.

I was just thinking about packing up and heading home when a student in my current class poked her head in and asked if she could talk to me.  I invited her in, assuming she had a question about the material for our class.  Instead, she says that she stopped by to say thank you.  She wanted to express her appreciation for me as a teacher–that she valued my style of teaching, the way I care about my students..and she gave me lovely card–a CARD!–with further thanks. How neat is that?!?!

In the research world, we get direct measures of our impact: through conference presentations, publishing, awards, and citation counts.  But in teaching, you may never know if you had an impact on a student.  They might take more classes with you, or they may switch their major, but how often do we get direct feedback that we made a difference in the life of a student?

It’s rare–or it has been for me–but I can tell you that this conversation yesterday is worth 10 APSR articles in my book.  It may not get me tenure, but it sure was life affirming.  Its 730 and I’ll be at work for hours yet, but the answer is certainly clear: Its definitely worth it.




Promoting reflection

One of the more interesting aspects that has emerged from this year’s running of my module on negotiating politics has been the problematic nature of reflection on the part of students.

At one level, reflection is front and centre for them, since their assessment is based entirely on a reflective piece they produce at the end of the module, on their experiences in class and its relation to theory and academic literature.  To bolster this, I have given them opportunities for formative work, and I discuss this aspect with them every single week.

And yet, at the end of each session I end up having discussions with students about what they are doing and why.  Yesterday’s class was structured around a disarmament game that I use to explore trust in negotiations: teams represent a provisional government, the national army and three factions.  The game also has a faction-of-a-faction, who were left at one side of the room and other teams could decide what to do with them, if anything.  For 90 minutes, this team just sat and watched, with only one team even trying to talk to them.  At the end of the game-play, we discussed what had happened and I pointed out that this team would have attacked everyone else if the agreement everyone else had reached was followed through.

I suggested to the students that simulations tend to be rather predictable, in that they usually only include what they need to, so if an ‘extra’ group appears, then it has a purpose, so non-engagement is probably not a good idea.  Likewise, when I had asked them some weeks ago to set up Twitter accounts for the session on communication, no one had apparently thought through why I might have asked them to do this.

I left them with the observation that next week’s game, on power, will last exactly 30 minutes: as we’ll see, in this case it’s almost impossible to work out why that information is important, but I hope someone will come and talk to me about it.  And if they don’t, then perhaps we’ll have another learning moment.

Color-Coded Grading

Following Chad’s post earlier this week about using rubrics, i thought I would share a method for grading papers that a colleague shared during a workshop.

Following the stoplight method of green is good, red is stop, and yellow is caution, this method involves marking up papers with colored highlighters to provide a visual cue to students as to their overall performance on the paper.  Sections where the student is doing well are marked in green; grammatical errors are marked in yellow, and substantive problems are marked in red.

The use of colors has a number of benefits.  The use of green balances the standard practice of using red ink to indicate problems, and using a separate color for grammatic problems allows us to note the problems without having to explain the problem in detail.  I find that it reduces the amount of time I spend making comments, as the color often suffices to indicate to students the nature of the problem or success without having to generate individualized comments. I find it particularly useful for the grammar comments, which I usually want to note but not spend a great deal of time on. This type of grading is also easily adapted for either hard copies (where you use actual highlighters) or electronic papers (using the highlighter in Word).

The color system is easily understood by students and only requires a few moments to explain when turning back the first paper.  I haven’t yet looked into whether this method works better for students than the traditional commenting method, but I do find that it reduces the amount of time I spend trying to generate comments for papers.

More on human nature

Following on last week’s discussion about when do people learn, as against their nature, I found myself after class having exactly the same conversation with a student.  The assessment for my module on ‘negotiating politics’ is a reflective piece, where students consider their development as negotiators, in light of both the literature and their own experience through the weeks.

Today’s class was about principled negotiation, taking Fisher and Ury’s classic “Getting to Yes” as the key text.  The student was bemoaning the idea that it was necessary to analyse one’s own actions and that over-analysis was the likely outcome.  Indeed, he claimed that now we’d covered the model, everyone in the class was just going to stick to it and no one would really learn anything.

I reminded him of the second round of Victor’s card game, where a student had just ploughed on because “it’s nice to have more than one card.”  I then also remarked that even though I’d closed today’s class with a reminder about the centrality of good (or indeed, any) preparation in being an effective negotiator, I was confident that not everyone in the class would prepare, not even for the session in a fortnight entitled “preparation in negotiations”, which I have flagged repeatedly as requiring preparation.

The student took the point, but it was apparent that he didn’t quite buy it.  And it’s here that the experiential model will really kick in.  In a few weeks, I’ll be able to go back to students and ask them how the theory they’ve learnt has helped them and shaped their actions; not because I think it will have changed much, but because it won’t.  Likewise, even though they know they should be keeping notes to help them produce their reflective piece at the end of the module, most will not have been particularly assiduous about it.  By making them confront their natures, I hope to make it more likely that they will change (or at least modify) them.

Then again, I’d not bet on it…

The Advantage of the Long Course

Last night my 8 week US politics course finished up–its the same credits as a 16 week course, but we meet once a week for 4 hours for 8 weeks, so everything is condensed. These types of courses are primarily aimed at the non-traditional student who works during the day, although many traditional aged students take them as well.

Three or four hour courses are not uncommon in colleges today, and when we tell job candidates about them, the reaction typically varies from mild horror to ‘thanks, i’d rather not have a job right now’.  My reaction was similar at first, but I’ve come to love these courses.  Here are my top five reasons to perk up at the chance to teach a long night class.

5. It counts as part of your load.

We have a 3-3 teaching load, with no more than 25 students per class.  The 8 week class counts as part of the load–which means that the other 8 weeks, I only have two courses.  Teach two night classes a semester, and you either have a 2-2 the entire year, or if you teach them at the same time, the odd 3-1.  And if you want to pick up some extra cash via an overload, a night class in your ‘off’ term is the way to do it.

4. You Can Take Advantage of the Down Time

You have to break up the time anyway–the best lecturer in the world is not going to keep students on their toes for 4 hours week after week. Sure you can give them ‘breaks’ but why not take it further?  In the middle of each class I give the students a 15 minute break followed by ‘current issues discussion’.  The break is long enough for them to scour the local food options for dinner; then we sit down and eat while talking politics for 20-30 minutes.  Each week 3-4 students are responsible for choosing the issue and leading the discussion, which has the added bonus of giving me a break and time to eat my own dinner. Its a great way to take a ‘break’ from the class while still giving the students time to talk and learn.

3. Active Learning Bonanza

Four hour classes offer an amazing opportunity to try out active learning principles.  There is so much room for simulations and games in these classes–finally you have the time to brief, play the game, and debrief.  You can get the students out of their seats and moving around, and still have plenty of time to hit the highlights of the week’s topic.  In a four hour class, active learning becomes a necessary part of the course.

2. Students Can Really Learn The Material

I don’t find any difference in student learning in the 16 and 8 week version of my US politics course.  As previously posted, I use the US citizenship test as a pre- and post- test in this class, and the 8 week students do just as well as the 16 week students (last nights crop was no different: on the pre-test only 1 student passed; on the post-test everyone passed).  Students sometimes feel that they are missing out by only having 8 weeks instead of 16, and indeed they have to learn the material twice as quickly.  But I can be confident that these students aren’t being cheated in any way.

1. They Push You as a Teacher

I may feel exhausted both before and after I step into the classroom, but while I’m there, I’m completely energized. Four hours isn’t a hardship: its an opportunity–an opportunity to figure out the essential ideas of each content area and how to deliver it to the students.  I can experiment with new types of assessment, with new games and simulations, with readings, and with different technologies.  The challenges of this kind of course have led me to engage in more reflection on my teaching than any other course.

Are there downsides?  Sure.  Its exhausting.  The students are sometimes late to class due to traffic or work or kid issues; and sometimes they power down despite your best efforts.  But these hassles are no different than those faced in day time classes.  So my advice?  If you get a chance to teach a non-traditionally timed class, jump at it.  You may be surprised at how much you enjoy it!



Leaving the Room Open to Unexpected Learning

Several weeks ago while playing a few rounds of the prisoner’s dilemma I came across an opportunity to learn from my students.  The game went in this way: Students were to play one round of the Prisoner’s Dilemma (rat out your partner in order to win more, or cooperate to reduce a prison sentence).

A pair of women in the corner simultaneously cooperated.  For those unfamiliar with the Prisoner’s Dilemma, the standard of the game is for both people to rat out their partners.  Sometimes people do cooperate but normally they don’t.

We discussed the gave at some length and then I began the standard social scientific move to iterated repeat games.  I asked the students to play 9 rounds of the game. After 9 rounds the two african american women were still cooperating though they had never talked before in class prior to this game.

Here is where it got interesting.  I explained that in general we should expect people to cooperate some but not all the time.  The students nodded along.  The two women in the corner looked at me quizzically. So I knew there were potentially different explanations for why cooperation and defection were good or bad strategies.  So I asked the students: “ladies, why were you constantly cooperating?” Another student, a while male, tried to explain to them how they were being irrational.  That they needed to realize that they weren’t maximizing their opportunities.

And one of the pair said, “You people don’t really know what it is to grow up in an environment where security really is a problem.” She said: “In my part of town you don’t cheat people or rat them out.  You can’t, no matter how much money you might get from it, because you don’t know if the person you cheat is going to show up and shoot up your mom’s house, or kill your brother.”

The classroom recoiled in horror.

I could have shut this down.  Clearly we had entered a space in which the student offered up a lived experience to question the theoretical conclusions. A lived experience that ran close to being socially, economically, and racially charged.

It is at these moments that we must decide whether to let it ride or to embrace it and try to dig deeper into it.  At once, here were two women presenting the room with an entirely different perspective.  One that most had never been privy to.  The learning was shifting immediately beyond the theoretical importance of the Prisoner’s Dilemma, to understanding a diversity of perspectives, to how to think about who was in the room, to try to consider what truth and the imperatives of action really dictated.

I was reminded that active learning isn’t always planned learning.  Control is the purpose of the rules in a game, but what exactly we are trying to control is the question.

My challenge to reader is to consider the way in which control is leveraged in active learning, and to question just how much we really need and for what purpose.

My thoughts keep going back to that awkward silence and the exasperation of the male student toward the continued cooperation of the two females. It was such an eerie and powerful thing to say….so much so that I’m not even certain what we learned exactly, and I suspect that this is the point of exposure to new perspectives.  Not so much that we come out with clear conclusions, but that we throw them into a controlled disarray.

A is for ‘awful’ or ‘awfully good’?

One of the joys of the teaching process is that you tend to get your feedback at a point when you can’t really do anything with it until the next time you run the class. You might argue that students get the same deal, but that’s another matter.  So here’s a quick and easy way to do some mid-stream modifications, using an “A-B-C” exercise.

After a few sessions of your class, when things have bedded down a bit, take 10 minutes to do this.  Give everyone in the class 3 post-it notes (other brands are acceptable).  Ask them to put one idea (anonymously) on each note as follows:

  • One note on something that they want to Abandon in the class;
  • One note on something that they want to Begin;
  • And one note on something they want to Continue.

Once they’ve written them, they can post them on the whiteboard/blackboard/wall.  With all the notes, you can then do a quick review with them, grouping similar points together and giving your first impressions.

The next class, you need to come back with some constructive feedback, to show that it’s not just been an exercise in raiding the stationery cupboard. Think about what’s reasonable to change, both in terms of effort and benefit, and about the reasons for not changing something (rather than just dismissing it out of hand).

This all works surprisingly well, it’s timely and it shows students that you do listen to their constructive input.  Even if it doesn’t result in big changes, it’s still a valuable group-building technique.

The only word of caution is if you have a class that is not working in some major way: because this is quite public and open as a process, it might cause more instability than it solves.