What to do about grading

This guest post comes from Eric K. Leonard, Shenandoah University

I had the pleasure of spending last week at the annual International Studies Association conference in Montreal.  It was a pleasure because I got to hear so many interesting discussions on pedagogy and honestly, ISA wasn’t always that way.  I attended my first ISA meeting in 1997 and the number of pedagogical panels was limited, to say the least.  

But as the years (actually decades but that makes me sound really old) have progressed, ISA has truly embraced the push by members to create space for pedagogical discussions at the annual meeting. I know for me personally, it has made attendance at this conference a joy. 

Amongthe plethora of pedagogical workshops, roundtable discussions, and panels that I attended one topic seemed to keep emerging – the assessment of students or grading. And within that topic, there was a lot of buzz around the idea of “ungrading”.  

I will admit that I generated some of the buzz since I am a vocal advocate of this method.  But in several discussions, this topic was raised without my prompting, albeit not always with the same enthusiasm and positivity that I hold for ungrading.  One thing was shared by most in the room and that was a disdain for grading. In one session, Simon Usherwood even mentioned that his appointment does not require grading!  Frankly, based on the level of jealousy that erupted from his confession, I was surprised he made it out of the room in one piece. So maybe we all can’t stop grading, but what would it mean to start ungrading? 

First, let me state that I am not a fan of the term ‘ungrading’.  As one of my students pointed out, they still get a grade at the end of the semester, so is it really ungrading? I much prefer self-reflective grading or something less clunky (suggestions are welcome). 

Second, ungrading is not one method of assessment.  It entails a lot of different methods that include specification grading, labor-based grading, minimal grading, authentic assessment, and others. The commonality amongst these methods is that students primarily self-assess through a process of self-reflection.  

The method that I employ is often considered the most extreme form of ungrading because as the instructor, I don’t grade any of my students’ work.  Because I teach in a hybrid format that includes a flipped learning model (maybe a conversation for another post) students submit a written assignment every week of the semester.  However, I don’t put a letter grade on those submissions.  

Instead, I provide students feedback to assist in their improvement of future submissions and to push students in terms of how they think about the week’s topic.  So throughout the semester, students never receive a letter grade.  But my university requires that I input a grade at the end of the semester.  

The way in which I ascertain student grades is through a process of self-assessment.  At three different points in the semester, students complete a self-assessment of their learning process.  This self-assessment includes questions about what they learned, what sparked curiosity, and questions about attendance, the number of assignments missed, if any assignments were late, participation level and quality of their comments during our classroom discussions, the effort a student has put forth, and the overall quality of their work, which can be compared to an “Answer Key” that contains examples of exemplary work from students in the course.  After students consider all of these different variables, they are asked to submit a grade and justify that grade.  

I explain to students that in most instances, I either accept that grade or input a higher grade.  And historically, that is the case.  However, I reserve the right to request a conference with a student if I am struggling to see how that student justified their grade.  In my experience, these conferences are rare (less than 1% of students per semester) and typically conclude with the student gaining a better understanding of the self-assessment both in terms of its criteria and its purpose.  In two years of ungrading, I had one student that refused to rethink their grade after a conference and therefore received a grade that I struggled to justify.  

With that brief accounting of how I employ ungrading, let me remark on a few critiques that typically emerge around this form of assessment.  

The first is that students, especially certain demographic groups, will simply overvalue their performance. In other words, students won’t truly self-reflect on their work and they will just provide themselves with a higher grade.  

I had a similar fear when starting this process but I can honestly say this has not been the case.  As mentioned above, only one student was adamant about their earned grade, despite my pointing out statements in their own self-assessment that didn’t seem to square with the perceived grade.  Because my policy is to let students grade themselves, I allowed that student’s perceived grade to stand.  But as mentioned, this has happened ONCE in two years.  Ballpark number, that is 1 in 250 students.  And a quick aside, that student would not have fit the demographic that the literature tells us would contest their grade. 

Second, as one participant stated during a panel at ISA, ungrading is “fake.”  I think what they meant by this statement is that ungrading does not accomplish what it claims to.  

An incomplete list of ungrading claims is that: 

  • Ungrading relieves the stress of grading and promotes greater learning rather than a focus on the grade.  
  • It encourages greater collaboration rather than competition among the students and between the professor and the students.
  •  It recognizes the subjective nature of traditional grading, especially in essay submissions, and allows students to agency in their learning process.

Given the context of the conversation, I believe the “fake” comment was directed at this final claim.  But what claims like this one fails to accept is that when an instructor grades an essay, research paper, or dissertation, there is a subject component to it and the student has no agency.  Thus, that is also fake.  

If we provide students the opportunity to be part of the discussion on their grades, we might get more thoughtful students that are more interested in learning rather than their GPA.  And although they do not have complete agency over the grading process, since I am creating the self-assessment form and providing an understanding of what exemplary work looks like, the level of agency is much higher than in a traditional assessment format.  I don’t understand why people see this as a dichotomous relationship in that students either have agency or they don’t.  The reality is that any increased level of agency for students in the educational process is a good thing and I believe ungrading serves to fulfill that goal.     

One final thought on ungrading.  And this is probably the most important point I can make in this post. 

Ungrading is not for everyone.  

Just like any other pedagogical method, ungrading has to fit with your teaching style, your approach to higher education, and your students.  Thus, this post is not an attempt to advocate that everyone adopt ungrading and to chastise those that refuse. Instead, it is an attempt to shed light on how one professor implements this assessment method in his classroom.  

You will notice that I didn’t discuss in any detail the literature on why we should stop grading or how it benefits the learning process.  This literature exists (in abundance) and engaging it might persuade some to give ungrading a try, but the question that every instructor has to ask themselves is whether they think this could work for their pedagogical approach and their students.  I answered that question with a resounding yes (and I have the student comments to support this claim)!  But that doesn’t mean your answer is the same.    

Notes from a conference III, or: why you hate grading

It’s probably bad to write three posts, one straight after the other, but hey: tough.

We’ve got more grading/assessment-related posts coming from people more on it than me, but a couple of times in the first days of ISA, we’ve had the classic teaching-related panel moment of everyone nodding along to ‘but everyone hates grading, right?’

[Parenthetically, one of those moments led to talk of how good it would be if our jobs didn’t including grading at all, whereupon I noted my job doesn’t include it, to many jealous looks. I’m mostly happy that I have so little grading to do (like a dozen scripts a year little), but part of me does miss it, mainly for the reasons set out here.]

The obvious question that no one really seems to keen to investigate is why we all seem to hate grading.

After all, it’s the main mechanism we have for evaluating student learning and, by extension, whether our teaching is getting the results we intend. It’s more rigorous than a chat at the end of class about learning gain or the impressions we glean from observing debates in the classroom, so why are we so down on it?

Let’s consider the usual suspects.

First up, ‘it’s repetitive’. Sure, if you have a pile of a hundred essays on the same topic, that is super-dull, especially if there’s a lot of overlap in the intellectual sources your students used (your lectures, the core textbook, the readings you suggested): it’s a rehash/mangling of what you’ve taught them and it’s probably not pretty.

So don’t make that your assessment. Give them choice, make them responsible for finding their own sources, get them to bring something more personal and individual.

My old negotiation course asked students for their only assessment point to tell me what they had learnt from the course: I was emphatic that it could be absolutely anything, as long as it was grounded in what had happened in class and was connected out to whatever literature they had found useful. Result was such a rich variety, every time I ran that course.

Second up, ‘it takes such a long time.’ Partly that’s about the time-elongating/mind-numbing of reading the same stuff a bunch of times that we just discussed, but partly it’s our demands of the process. Often, we normalise the essay of X pages or Y thousand words as proof of learning, even as we rool our eyes at ‘all that text’.

So go for concision, not length. You know it’s harder to produce a good short text than a good long one: the demand to focus much more tightly on what’s important, to lose the blah, is something we find tricky (to judge by the number of ropey abstracts you and I have read). Set a much shorter length piece, or ask for a 2-page policy brief, or a poster, or an infographic, or a 3 minute video.

Of course, shorter assessment outputs doesn’t automatically mean quicker grading, but it helps. As does trying things like videoing your feedback, or asking students to make their own evaluations of their or their peers’ work.

Finally, ‘students find it boring/unpleasant (and so do I)‘. Find me the student who loves to do assessments and I’ll just start wondering what happened to them to make it come to this. I get that everyone thinks it’s an imposition and chore.

So involve your students in finding assessments that engage them more/turn them off less. Amanda’s breakfast exercise for research methods is my go-to here: super-fun, super-engaging allowing them to really capture their experiences in an assessment.

If assessment is about cementing the bond of teacher and learner, then why wouldn’t you want to bring them in to find a mutually-satisfactory assessment regime?

Unless you’re working under a highly prescriptive institutional system, you hold the power to make your assessment and grading less boring. So use it.

Notes from a conference II

Probably a learning moment here to, but more just a reminder that we are an award-winning blog

Apart from my worries, ISA is also making me think about the purpose of teaching.

I’d been thinking about this anyway, partly because of this post from John Warner that popped up on my timeline recently.

The instrumentalisation of education as a stepping stone to the next thing, working, has been around for a long while now and it is pernicious in its compromising of student engagement: what they are told they need is a good grade, not a good education and given the incentive structure we shouldn’t be surprised if they act accordingly.

I’m guessing you’ve had these students in your class, more or less often. Either the ‘just tell me what I have to do to get a good grade’ version or the ‘my corporeal self is present, but not my mind’ version.

Regardless of whether we cleave to a knowledge transmission model of teaching, or (more likely among our readership) an active learning approach, you still rely on the student being present and engaged.

So what to do?

One of the conversations I sat in on here spoke about teaching as problematisation; making things uncertain to open up space for reflection, debate and development.

I like that idea. We aren’t here to solve the world for our students, but to invite them to create their own understandings of situations and problems, to recognise and act on their agency in that world.

But again, if our students aren’t in the metaphorical room, what good is challenging conventionalities?

My reflection is that much of what I have done over the years is try to distract students from their learning.

Using games and simulations, taking the class out onto the campus lawns, literally removing the physical and conceptual constraints of ‘the class’: it’s all grounded in a notion of ‘I’m not expecting this: what is it?’

The disruption of the moment hopefully leads to the delight of doing something different, and that hopefully leads further to ‘dammit, I’ve been learning stuff’.

Colleagues here have spoken about how alums tell them how they remember the random moments – the simulation, the talk of Russian hookers, the exercise with the blindfold – long after they forget the ins and outs of whatever IR theory was on the syllabus that week.

It’s why I will always argue that those who say fun can’t be serious are wrong.

Precisely because fun isn’t (linguistically) serious, it opens up the possibility of serious learning, of helping students to gain insight from all of their interactions with the world.

That’s why I can learn from the movie I watch on the plane, why I can learn from watching my child play with Lego, why I can learn from a big erection (you’re not clicking that link, are you?): because all these things are surprising moments to learn. The only difference is that I’m consciously looking for that.

Our present-not-present students are not, but that doesn’t mean we can’t keep on challenging their assumptions about what ‘learning’ is and keep on offering them cause to think.

It’s not foolproof, it’s not a comprehensive solution, but it’s a way in.

Notes from a conference I

Also worried about whether the duck can cope with that harsh New England climate

We’re a bit more than half-way through ISA here in snowy/rainy Montreal and I’m concerned about a number of things.

Most mundanely, I’m worried that even as we’re getting people interested in writing some guest posts for us, the top post is about the movie I watched on the way over.

Almost as mundanely, I’m worried that I’m not built for such an extended period of rich dining.

But mostly, I’m worried about how I teach.

There have been lots of roundtables and discussions about teaching at the conference, touching on all aspects of our practice. It’s been a while since I’ve had so much time to reflect and debate these points, in the broad way that we have been.

Obviously, much of this has been couched in the language and frame of the US experience, with its particular concerns and touchstones. But abstracting from that isn’t so much of a problem as the nature of my work.

As regular readers (who are more numerous and engaged than I realise: props to you all) will know, I work in a distance-learning University, where I never interact directly or synchronously with my students.

Much of our discussions have been about building the relationship with the students in our class, to create a space in which to exchange and learn, however we might specifically go about that.

But I don’t have that. Instead, I have to imagine my students.

Imagine who they are, how and why they study, what they know and what they want to know. My colleagues and I take an educated guess and pitch our materials in that general direction.

And then some (long) time later we find out how those students did on assessments, get some feedback and tack for the next time around. Not that the next time will necessarily be the same types of students.

I’d not really thought about this until now, mainly because I know that the system works pretty well in practice: student performances are strong, feedback is positive, our institution’s reputation is sound – none of that would be the case if we weren’t getting it right most of the time.

But the general is not the particular: just because we got it right before, doesn’t mean I will get it right now.

My worry is perhaps the worry we all hold, whatever our situation – that there is a risk in teaching, for our students, for us, for the process.

That worry is also perhaps what keeps us motivated to be thoughtful about what we do. By turning our worries into action, we might forestall the failure, we might improve our chances of making a difference, we might help our students to be, well, just more than they were before.

So I’m owning my worry, because my worry is pushing me forward, not holding me back.

And it shouldn’t hold you back either.

Don’t think, just do: Why Maverick is a bad teacher

Yes, I’ve been sent across the ocean once again, this time to ISA in Montreal.

We’ll be talking about teaching here (although not as much as planned after the cancellation of the pre-conference day) so I thought I should use the flight over to bone up on my movie representations of teaching.

As I’ve previously discussed, good teaching is apparently beyond any film producer, so I didn’t approach Top Gun: Maverick with much confidence. The film has already stimulated a pile of thought pieces about what it says about the US’s confidence in global leadership (spolier: not much positive), but no one has recognised the insights that Maverick has to convey about the practice of pedagogy.

[We’ll pause here to say that I probably have lots of spoilers hereafter, but frankly, if you didn’t already see the film, then you’re not going to now, so tough]

At first blush, this is a movie all about active, experiential learning: the team have to discover for themselves how to perform the seemingly impossible task, with Mav as some kind of sage mentor to help them believe in themselves. There are threshold learning moments, as pilots find out what more is possible when they do things like, um, go faster or, err, disobey orders. Hence the recurring motif of ‘don’t think, just do’.

There’s also a heterodox spin here: rule-breaking here is the goal, to help get inside the enemy’s OODA loop. ‘It’s not the plane, it’s the pilot’ comes up quite a lot and the unpredictability of the latter is the key to defeating SAMs and 5th-Gen fighters.

But all this falls down on the conceit that Maverick himself is the best there is: his excellence is the key to unlocking his students’ potential.

He is the one who has to show that the timed bombing run can be done within the time, that the g-forces can be withstood, that the fractious individuals can be knitted into a team through a session of beach ‘football’.

Yes, this is a very particular kind of teaching situation, with a very specific learning objective, but leaves us with the trope of teaching-as-performance, where students are to be stunned into learning by the sheer majesty of what they see before them.

If nothing else, most of us aren’t those kinds of people.

And nor should we be.

Active learning is about helping students to see the doors to learning and getting them to open those doors and walk through: not smashing them down then showing students the limits of what can be done beyond them.

The objective is not to be the hero of the classroom, but its facilitator, because the classroom is about the learners, not the teacher.

Which is why I always like coming to conferences like these, where I can learn from others who’ve done stuff that I haven’t and we can reflect on how to move things forward.

[I also watched Tom Cruise in Edge of Tomorrow on the flight, which I now realise also involves a lot of teaching, but you really don’t want my opinions on that one]

Performing as academics

What’s your evidence base for this?

I had the too-rare pleasure to chat with an anthropologist last night.

We are both part of a big European-funded project that is having its final conference in Brussels this week, with a focus on connecting to policy-makers and other practitioners in the field of refugee protection.

It’s a big deal, for me as much as anyone: I’ve been leading on the conference planning for the past 18 months as we bring together speakers from at least three continents and juggle a very large number of moving parts.

The conclusion of the first day has shown that planning does pay off, from bringing some marker pens that got used within ten minutes of arriving to thinking about the structure of the sessions.

Let’s focus on the second of these elements here, important though ready access to marker pens might be.

From the start, I was clear with myself that I didn’t want this to be an academic conference.

By that I mean the format you know from pretty much every single big conference you go to: 90 minute sessions with 4 or 5 papers, hopefully not just read out in a monotone but quite possibly, with 5 minutes for someone to make more of a comment than a question.

The world has plenty of those kinds of things already and this project does not need to add to the pile.

Our work has a very clear and evident policy application: studying refugee protection in Europe and beyond was already a big thing three years ago when we started, even before the effects of covid or the war in Ukraine shook things up even more.

So I wanted to break out of the mold as much as possible.

My initial suggestion that we should just have an afternoon where we sat in an informal space and attendees came up to chat with us was not popular, so we rowed back to the format we have now: some keynotes, some roundtables with significant non-academic presence, and lots of time for chat/networking.

In broad terms, this seems to be working in breaking down barriers between academics and practitioners: our lead investigator did a great job of opening the event with a clear single message about what our work shows to practitioners, for example.

But it’s also hard.

At points, we drift back into the comfort of roles – and this is where the anthropologist and I found common cause – rather than challenging ourselves to push back and to make things work for us.

This isn’t even so much about trying not to talk in jargon or to argue on precise points of definition (although also that), as it is about falling into classroom modes, with lecturers and students, each doing what they are supposed to.

If we want to build more effective and significant links from academia into other communities then we have to be mindful of this; we have to try really hard to change the incentives for our colleagues to adapt appropriately.

That means more work on formats, more challenging of assumptions, more reflection about the specific needs of others.

I’m taking this conference as a win even before it ends, but I know I could do better and next time I will try harder.

Getting your bearings

Not in full-time education, bu still interested in learning

My household is full of people involved in education, either as students or as teachers. However it’s only this academic year that my (much) better half has moved into a regular teaching role, after years of research-only activity.

It’s been great to see how she has moved from anxiety about what to do to reflection on how best to make her classes work and how it all marries up to her research activity, not least because it’s also given me an opportunity to reflect on what I do too.

Indeed, it was this process itself that was perhaps the most stimulating, since it highlighted one difficulty for a new teacher, namely what – practically – to do.

Training programmes are typically on-the-go, using your classroom experience to stimulate reflection on pedagogy and learning and theory-level stuff. But how to get to a point of being confident about stepping into that classroom in the first place?

What seems to be missing is a simple ‘how to’ guide that offers practical advice on really mundane stuff, like dealing with questions, or designing your first lecture, or what to keep in mind for your first session.

All the stuff you have internalised from your teaching, basically.

Looking around, I don’t see a resource like this, which surprises me (and also makes me wonder if I’m just looking in the wrong places): we all have to start somewhere.

In my case, a long time ago, it came from a lot of watching people teach/try to teach me and reflecting on what did and didn’t work. But that’s not a normal path, I’m aware.

Maybe you came to this some other way, in which case I’d love to hear about it, or indeed anything like the resource I describe.

This comes back to some of the themes we touched on in Baltimore, namely lowering costs to entry to better pedagogy: it’s as true for ‘basic’ stuff as it is for fancier approaches like active learning. One reason why teaching doesn’t have the same rep as research is precisely that we don’t institute the same level of training in the latter.

But that might another post for another day.

Really? Actually really?

Spangly

Some more time to digest the lessons of Baltimore’s TLC, and I’m coming back once again to a big theme (for me) about how we use simulations and games.

At several points during the conference, I found myself pondering how we reconnect sims and games back into the rest of our teaching practice, or even whether we do that at all.

The argument I’d make (and I made) is that sims are essentialisations of the world: they take some aspect – an event, a relationship, a dynamic – and places it front and centre, stripped of all the other stuff so we can see it more clearly for our edification.

That essentialisation gives us the core of the gameplay of the sim. The mechanism we want to focus on logically also informs the choice of how we focus on it. So if we want to understand voting dynamics, we build a sim with voting in it, plus whatever elements we want students to consider as factors.

None of this should be controversial – it’s how all teaching works, in practice. We draw attention to something, to allow us to think about, discuss and learn about it.

Sims are just a more overt example of this.

But that essentialisation also implies a need to have an active reconnection, post-activity.

It’s why we talk so much about debriefing: we are trying to draw lessons out of the activity, back into the other modes of teaching we use, both within a course and more generally.

By definition, active learning places the locus of learning within the student, so to know what they have learnt, we (as teachers/instructors/facilitators) have to work to pull that out of them.

But what does that involve?

Typically, when we talk about debrief we are looking at ‘what did you learn?’ type questions. Asking students to vocalise their understandings and experiences in a more-or-less structured manner.

Obviously here, we can cover all the usual bases of substantive knowledge, practical skills and broader socialisation into communities.

However, the whole essentialisation process means we also need to ask a different type of question too: ‘how did this experience match up to the actual thing you’re studying?’

One of the first papers I wrote on sims was about realism and the ways we can make our sims versimilitudinous [sp?], but sims are always approximations to reality and we need students to be both aware of and thoughtful about this.

In essentialising, we strip away ‘the other stuff’ because it’s not that important to our objective. But ‘not that important’ doesn’t mean ‘unimportant’.

So part of our debriefing needs to ask students whether and how the simplifications of the sim distort our understanding of what matters. To go back to that voting dynamics exercise I mentioned earlier, if we exclude a particular element, do students now think that element isn’t relevant?

One way I’ve tried to explore this is being asking students to come up with ways they could make the sim work in a more realistic manner: and often I’ll use that the next time out. This process invites either constructive additions or thoughtful reflection about why something can’t be included (and the impact thereof).

None of this is to say that sims aren’t any good; only that they are imperfect tools (like all pedagogies) and so we have to ensure that students recognise and reflect on that.

The persistent gap in simulation design literature

About eight years ago, I wrote the following:

….there remains a severe lack of basic ‘how-to’ guides (whether grounded in higher pedagogic theory or not); instead, the typical process of spreading use of simulations is by word of mouth and the adaptation of existing models to new situations. 

I was arguing that while there was a metric tonne of individual case studies of the ‘show and tell’ type and another pile of more theoretical musings on the nature and purpose of simulation games, there was a gap in-between.

The theory pieces were great for that paragraph at the top of your paper for justifying ‘why simulations’ but didn’t tell you how to go about actually doing one, while the ‘show and tell’ stuff sometimes gave you enough to re-run that particular sim, that sim was never quite what you needed.

No tools, no easy way forward.

All of this came flooding back during TLC in Baltimore this last weekend, with many of the colleagues on the track talking about starting out on their sim-building journey and looking for help.

Even I, with a pretty healthy dose of experience, found that when I was designing an online, asynchronous sim, there was basically nothing at all that worked through the options and mechanics. If you read my paper, you’ll see that much of it ended up being a working from first principles.

This persistence of a lack of operational literature is frustrating.

It’s also quite understandable, in that there aren’t that many people who are bothered enough not just to build their own sim, but also to abstract from that out to general design and implementation principles.

And yes, before you ask, I did write various things, gathered here. But that’s not really enough.

Maybe you need something like Classroom Assessment Techniques, the classic handbook from Cross, or maybe you need something else more dynamically and interactively built online, but as long as we lack these kinds of resources then we’re doomed to lots of people spending lots of time they don’t need to, reinventing wheels that have been reinvented many, many times before or – and worse -just giving up because it’s too tough.

As usual with such thoughts, while I can identify the problem, I can’t offer a easy solution.

Maybe you can, in which case I’d love to hear from you. As would many others.

#TLC2023 prep time

I open by freely admitting that this post is totally self-serving, but since it’s the first time in ages that I actually wrote the paper ahead of the conference, I’m going to extol the virtues of writing your paper ahead of the conference.

APSA’s TLC is nearly upon us and the heritage it brings of a track model means that pre-written and -circulated papers is still very strong. You can check out the submissions so far here.

As an opportunity to get focused and specific feedback on your work, especially if you’re looking to publish, this is pretty hard to beat: getting a bunch of people working in the same field, in a room for most of two days, sharing ideas, isn’t something you get to do that often in conferences.

And if, cough, one of those people is an editor on a leading L&T journal, then all the better.

Having already started to work my way through the papers on my track, I can see we’ll be having a lot of very productive debate in Baltimore, which we’ll be doing our best to share with you all.