What to do about grading

This guest post comes from Eric K. Leonard, Shenandoah University

I had the pleasure of spending last week at the annual International Studies Association conference in Montreal.  It was a pleasure because I got to hear so many interesting discussions on pedagogy and honestly, ISA wasn’t always that way.  I attended my first ISA meeting in 1997 and the number of pedagogical panels was limited, to say the least.  

But as the years (actually decades but that makes me sound really old) have progressed, ISA has truly embraced the push by members to create space for pedagogical discussions at the annual meeting. I know for me personally, it has made attendance at this conference a joy. 

Amongthe plethora of pedagogical workshops, roundtable discussions, and panels that I attended one topic seemed to keep emerging – the assessment of students or grading. And within that topic, there was a lot of buzz around the idea of “ungrading”.  

I will admit that I generated some of the buzz since I am a vocal advocate of this method.  But in several discussions, this topic was raised without my prompting, albeit not always with the same enthusiasm and positivity that I hold for ungrading.  One thing was shared by most in the room and that was a disdain for grading. In one session, Simon Usherwood even mentioned that his appointment does not require grading!  Frankly, based on the level of jealousy that erupted from his confession, I was surprised he made it out of the room in one piece. So maybe we all can’t stop grading, but what would it mean to start ungrading? 

First, let me state that I am not a fan of the term ‘ungrading’.  As one of my students pointed out, they still get a grade at the end of the semester, so is it really ungrading? I much prefer self-reflective grading or something less clunky (suggestions are welcome). 

Second, ungrading is not one method of assessment.  It entails a lot of different methods that include specification grading, labor-based grading, minimal grading, authentic assessment, and others. The commonality amongst these methods is that students primarily self-assess through a process of self-reflection.  

The method that I employ is often considered the most extreme form of ungrading because as the instructor, I don’t grade any of my students’ work.  Because I teach in a hybrid format that includes a flipped learning model (maybe a conversation for another post) students submit a written assignment every week of the semester.  However, I don’t put a letter grade on those submissions.  

Instead, I provide students feedback to assist in their improvement of future submissions and to push students in terms of how they think about the week’s topic.  So throughout the semester, students never receive a letter grade.  But my university requires that I input a grade at the end of the semester.  

The way in which I ascertain student grades is through a process of self-assessment.  At three different points in the semester, students complete a self-assessment of their learning process.  This self-assessment includes questions about what they learned, what sparked curiosity, and questions about attendance, the number of assignments missed, if any assignments were late, participation level and quality of their comments during our classroom discussions, the effort a student has put forth, and the overall quality of their work, which can be compared to an “Answer Key” that contains examples of exemplary work from students in the course.  After students consider all of these different variables, they are asked to submit a grade and justify that grade.  

I explain to students that in most instances, I either accept that grade or input a higher grade.  And historically, that is the case.  However, I reserve the right to request a conference with a student if I am struggling to see how that student justified their grade.  In my experience, these conferences are rare (less than 1% of students per semester) and typically conclude with the student gaining a better understanding of the self-assessment both in terms of its criteria and its purpose.  In two years of ungrading, I had one student that refused to rethink their grade after a conference and therefore received a grade that I struggled to justify.  

With that brief accounting of how I employ ungrading, let me remark on a few critiques that typically emerge around this form of assessment.  

The first is that students, especially certain demographic groups, will simply overvalue their performance. In other words, students won’t truly self-reflect on their work and they will just provide themselves with a higher grade.  

I had a similar fear when starting this process but I can honestly say this has not been the case.  As mentioned above, only one student was adamant about their earned grade, despite my pointing out statements in their own self-assessment that didn’t seem to square with the perceived grade.  Because my policy is to let students grade themselves, I allowed that student’s perceived grade to stand.  But as mentioned, this has happened ONCE in two years.  Ballpark number, that is 1 in 250 students.  And a quick aside, that student would not have fit the demographic that the literature tells us would contest their grade. 

Second, as one participant stated during a panel at ISA, ungrading is “fake.”  I think what they meant by this statement is that ungrading does not accomplish what it claims to.  

An incomplete list of ungrading claims is that: 

  • Ungrading relieves the stress of grading and promotes greater learning rather than a focus on the grade.  
  • It encourages greater collaboration rather than competition among the students and between the professor and the students.
  •  It recognizes the subjective nature of traditional grading, especially in essay submissions, and allows students to agency in their learning process.

Given the context of the conversation, I believe the “fake” comment was directed at this final claim.  But what claims like this one fails to accept is that when an instructor grades an essay, research paper, or dissertation, there is a subject component to it and the student has no agency.  Thus, that is also fake.  

If we provide students the opportunity to be part of the discussion on their grades, we might get more thoughtful students that are more interested in learning rather than their GPA.  And although they do not have complete agency over the grading process, since I am creating the self-assessment form and providing an understanding of what exemplary work looks like, the level of agency is much higher than in a traditional assessment format.  I don’t understand why people see this as a dichotomous relationship in that students either have agency or they don’t.  The reality is that any increased level of agency for students in the educational process is a good thing and I believe ungrading serves to fulfill that goal.     

One final thought on ungrading.  And this is probably the most important point I can make in this post. 

Ungrading is not for everyone.  

Just like any other pedagogical method, ungrading has to fit with your teaching style, your approach to higher education, and your students.  Thus, this post is not an attempt to advocate that everyone adopt ungrading and to chastise those that refuse. Instead, it is an attempt to shed light on how one professor implements this assessment method in his classroom.  

You will notice that I didn’t discuss in any detail the literature on why we should stop grading or how it benefits the learning process.  This literature exists (in abundance) and engaging it might persuade some to give ungrading a try, but the question that every instructor has to ask themselves is whether they think this could work for their pedagogical approach and their students.  I answered that question with a resounding yes (and I have the student comments to support this claim)!  But that doesn’t mean your answer is the same.    

Notes from a conference III, or: why you hate grading

It’s probably bad to write three posts, one straight after the other, but hey: tough.

We’ve got more grading/assessment-related posts coming from people more on it than me, but a couple of times in the first days of ISA, we’ve had the classic teaching-related panel moment of everyone nodding along to ‘but everyone hates grading, right?’

[Parenthetically, one of those moments led to talk of how good it would be if our jobs didn’t including grading at all, whereupon I noted my job doesn’t include it, to many jealous looks. I’m mostly happy that I have so little grading to do (like a dozen scripts a year little), but part of me does miss it, mainly for the reasons set out here.]

The obvious question that no one really seems to keen to investigate is why we all seem to hate grading.

After all, it’s the main mechanism we have for evaluating student learning and, by extension, whether our teaching is getting the results we intend. It’s more rigorous than a chat at the end of class about learning gain or the impressions we glean from observing debates in the classroom, so why are we so down on it?

Let’s consider the usual suspects.

First up, ‘it’s repetitive’. Sure, if you have a pile of a hundred essays on the same topic, that is super-dull, especially if there’s a lot of overlap in the intellectual sources your students used (your lectures, the core textbook, the readings you suggested): it’s a rehash/mangling of what you’ve taught them and it’s probably not pretty.

So don’t make that your assessment. Give them choice, make them responsible for finding their own sources, get them to bring something more personal and individual.

My old negotiation course asked students for their only assessment point to tell me what they had learnt from the course: I was emphatic that it could be absolutely anything, as long as it was grounded in what had happened in class and was connected out to whatever literature they had found useful. Result was such a rich variety, every time I ran that course.

Second up, ‘it takes such a long time.’ Partly that’s about the time-elongating/mind-numbing of reading the same stuff a bunch of times that we just discussed, but partly it’s our demands of the process. Often, we normalise the essay of X pages or Y thousand words as proof of learning, even as we rool our eyes at ‘all that text’.

So go for concision, not length. You know it’s harder to produce a good short text than a good long one: the demand to focus much more tightly on what’s important, to lose the blah, is something we find tricky (to judge by the number of ropey abstracts you and I have read). Set a much shorter length piece, or ask for a 2-page policy brief, or a poster, or an infographic, or a 3 minute video.

Of course, shorter assessment outputs doesn’t automatically mean quicker grading, but it helps. As does trying things like videoing your feedback, or asking students to make their own evaluations of their or their peers’ work.

Finally, ‘students find it boring/unpleasant (and so do I)‘. Find me the student who loves to do assessments and I’ll just start wondering what happened to them to make it come to this. I get that everyone thinks it’s an imposition and chore.

So involve your students in finding assessments that engage them more/turn them off less. Amanda’s breakfast exercise for research methods is my go-to here: super-fun, super-engaging allowing them to really capture their experiences in an assessment.

If assessment is about cementing the bond of teacher and learner, then why wouldn’t you want to bring them in to find a mutually-satisfactory assessment regime?

Unless you’re working under a highly prescriptive institutional system, you hold the power to make your assessment and grading less boring. So use it.

The robots are coming! And they’re… writing essays!!

It’s a truism that no academic is actually interested in assessment. Sure, there are certainly academics who find assessment stimulating and and engaging as a topic, but none of us has ever met one.

Except you have: me.

I never really understood the antipathy towards assessing: maybe it’s a carryover from being a student, where being tested felt, in very large part, like being punished. And I’m not going to pretend that I really liked sitting down to go through scripts.

One big exception to that was the reflective piece that my negotiation students wrote about their work, exploring and expanding on what they had taken from the course and contextualising it in the wider literature. Every one was a fascinating insight into my students’ heads, in a way that 4,000 words on ‘IR is overly fixated on power’ never is.

Any way. The only point were colleagues do seem to get more interested is when they have a problem with their assessment.

Right now, that problem is ChatGPT.

If you’ve somehow missed this one, this is an AI text generator, capable of creating extensive – and seemingly well-written – responses to minimal prompts. Including to things like essay titles.

To say that colleagues have been concerned is very much an understatement.

Both online and in-person, I’ve seen colleagues describe the system as the death knell of the essay format in assessment. Text is both of a standard that it could satisfy criteria to get a passing grade and that isn’t going to trigger anti-plagiarism software (since it’s not cut-and-paste, but organically created).

For all the rumours that this latter software will become upgraded to pick up on such AI-generated text, the feeling is deeply pessimistic.

My own view is perhaps more measured, mainly because with all the examples I’ve seen I haven’t felt the output is that impressive, especially for any one looking to bypass they way to the kind of higher grade that so many students seek out.

As others have noted, the quality and rigour of such texts isn’t up to much, which means both that it’s possible to pick up on AI generation (even if evidencing a plagiarism case is still a massive pain in the neck) and that students get a dubious amount of return (in grade terms).

But the bigger point is that text is only part of how assessment works.

The choice of questions you ask and the requirements you impose on students also matter massively.

Take that negotiation class I mentioned. Because I knew all the students, saw all they did in class and debriefing them extensively at the time about their learning, I had a very clear idea of what might be in their reflective pieces.

So if someone tried to write about stuff they hadn’t done (and a couple tried), I knew and could mark accordingly.

More generally, this all should be making us think more carefully about what assessment is for. And part of that is acknowledging that the very large majority of students don’t want to cheat on their education: sure it’s less effort (they think), but it screws when they are out in the world, trying to use skills or understanding they don’t actually have.

So I leave you with this example of how we can get students to engage with these challenges. It’s not a whole solution, but it is a recognition that blind panic or utter despair aren’t helpful responses.

We’ll be coming back to this several more times, no doubt, but I’d love to hear your thoughts on the matter.

Ungrading

A short one today, to encourage you to read this thread from C Thi Ngiuyen on how he’s challenging students’ understanding of grading:

Suffice to say here that his ideas resonate a lot with my own, but he’s in a position to do more about it with his class. For those of you who are bound to your institutional requirements on grading and assessment, this is still a really useful discussion to have, with both students and colleagues.

Assessing large groups

Among the many upsides of working with relatively small groups of students for most of my career has been that I’ve not been driven primary by volume management concerns. I could look across at my colleagues in Law or Business and laugh at the thought of having to juggle several hundreds of exam scripts at once.

(One time, a Business lecturer proudly told us how he’d marked 400 exams in three days, only for it to become clear it was a multiple-choice paper, with answers scanned in, which raised questions about why it had taken so long.)

But in the spirit of cooperation, a recent tweet about the need to treat assessment as an integral part of our teaching activity prompted this response from a Legal colleague:

This is an issue for many of us at some point: the big compulsory course where innovation feels like a luxury.

So what do to?

Sylvia’s dilemma is three-fold: assessment needs to a) serve learning objectives, b) minimise opportunities for cheating, and c) be practical to turn around with reasonable speed. We’ve not had the chance to speak about her specific situation, so what follows should be read more generically.

My personal view is that we always have to place learning objectives first in assessment: do we test for the things that we consider it essential that the students should have learnt?

In any course or module that covers a variety of elements: substantive knowledge; research skills; presentational and interpersonal skills; and more general aspects of critical thinking and building confidence. That breadth is important, because it underlines that ‘knowing facts’ isn’t the be-all and end-all here: even for us in academia, we probably make as much use of the skills and competences we gained from our study as we do the knowledge (and we’re at the high end of a spectrum of knowledge use).

Sylvia mentions vivas as a nominally ideal form of assessment, possibly because it’s interactive and personal and offers lots of opportunities to test how far a student can work with what they know. Having sat through vivas for a course of 100 students, I could point up some issues, but the analysis still holds: here’s something that better serves the learning objectives.

So are there other ways to get that same benefit without the big time implications of a viva system?

Two strategies suggest themselves, if we’re treating final written exams as an unsatisfactory option: different formats and collaborative working.

Asking students to produce posters, infographics or podcasts not only opens up different ways of presenting material, but also requires considerable distillation of substantive knowledge into key points, which in turn stimulates more critical engagement. Yes, students will be unfamiliar with the practicalities, but this can be covered with some preparatory sessions, and it develops presentational skills that might be otherwise neglected.

If you want to stick to text, then asking for shorter written pieces – policy briefs, submission to a court – can also keep the focus on distillation, plus give experience in formats they might encounter in their work (unlike a long-form essay).

And all of these options could be used with collaborative approaches too. Learning to work together is a valuable skill [he writes from his shed], so why not test for that? Group projects can be marked for the group as a whole, plus with individual marking for short reflective pieces on what each person contributed and got from it.

Of course, free-riding is an issue, and some disciplines might encounter accreditation barriers on collaborative assessment, but the problems need not be any greater than for final exams.

The right answer will vary from case to case: your capacities; the nature of your course; your institution’s attitude; the willingness of your students to buy into it. But these discussions are still worth having. Just because things have ‘always been like this’, doesn’t mean they should continue like this, especially if it’s not working for you or your students.

If you have more ideas on this, or what to chat about your assessment, drop me a line and we’ll talk.

The interrelation between attendance and writing assignment in a PBL course

This guest post comes from Patrick Bijsmans (Maastricht University) and Arjan Schakel (University of Bergen)

In one of his recent contributions to this blog, Chad asks why students should attend class. In his experience

[C]lass attendance and academic performance are positively correlated for the undergraduate population that I teach. But I can’t say that the former causes the latter given all of the confounding variables.

The question whether attendance matters often pops up, reflected in blog posts, such as those by Chad and by Patrick’s colleague Merijn Chamon, and in recent research articles on the appropriateness of mandatory attendance and on student drop-out. In our own research we present strong evidence that attendance in a Problem-Based Learning (PBL) environment matters, also for the best students, and that attending or not attending class also has an influence on whether international classroom exchanges benefit student learning.

Last year we reported on an accidental experiment in one of Patrick’s courses that allowed us to compare the impact of attendance and the submissions of tasks in online and on-campus groups in Maastricht University’s Bachelor in European Studies. We observed that that attendance appeared to matter more for the on-campus students, whereas handing in tasks was important for the online students.

This year the same course was fully taught on-campus again, although students were allowed to join online when they displayed symptoms of or had tested positive for Covid-19 (this ad-hoc online participation was, unfortunately, not tracked). We did the same research again and there are some notable conclusions to be drawn.

In the first-year BA course that we looked at, students learn how to write a research proposal (see here). The course is set up as a PBL course, so it does not come as a big surprise that attendance once again significantly impacted students’ chances of passing the course.

Continue reading “The interrelation between attendance and writing assignment in a PBL course”

Learning about teaching

Blossoming

Last night found me at our kids’ school, for a talk on revising. Aside from being a reminder of how quickly people decide that facemasks aren’t prudent any more, it brought home some lessons about the way we construct teaching for others.

The talk was primarily a run-through of what will be happening after Easter with exams, plus subject-specific sections on useful resources and good revision practice. Its content was much as you imagine, and as familiar to me (as a teacher) as it was to my daughter (who’s now on her third time of hearing it all in as many weeks).

So what’s worth mentioning here, on a site devoted to university education, where we don’t (usually) draw parents into it all?

The teachers here have clearly had some training on revision, including some useful models of ‘how to revise’, which they brought to the table. But what was missing (for me at least) was an unpacking of how revision fits into the broader process of learning.

Back at the start of the year, we got a welcome talk about what the next cycle for our daughter would be (the two years up to her first major external exams). In that was lots of stuff, but not so much about how keeping materials and notes would be a key part of ‘revising’, in the sense that they discussed last night. Revision implies vision a first time, and all the revision techniques set out in the current talk require a baseload of substantive knowledge and understanding to be able to produce the materials for effective assessment performances.

Put the other way around, if you’d not done the work until now, having six weeks until the tests to revise as the school would like you to is not a viable proposition.

And this is where this all matters for you (and me). Assessment (and by extension, revision) is too often treated as a disparate element of the educational experience; something tagged on the end, just because we have to.

Instead, assessment is an integral part of learning and should be handled as such, a logical extension to what happens in a class and in a student’s broader package of work through a programme.

This disconnect was evident in a couple of other places too, last night.

One of the teachers asked that students didn’t come to them, asking for ‘help with something vague’, but rather with a precise and focused query: ‘I have tried to do this past paper question on topic X and I can’t seem to make sense of it, despite several tries’, seemed to be the preferred line.

Now, as a teacher, I appreciate that more precision means more scope to get into the nuts and bolts with a student, but I also appreciate that the bigger problem is students not coming to ask for help at all. If I were a student who was struggling, being told I now needed to come with a precise inquiry strikes me as more daunting.

Here the issue is one of assumption-making about student engagement and buy-in to the programme of study. Even the most wonderful teaching set-up does not guarantee that engagement and we always have to be alert to those students that haven’t found their place within it.

That’s best treated not as the fault of the student, or the teacher, but of the specific instance. In a university setting we have more discretion to change and adapt that instance to accommodate individuals on a different path, but in a much more prescriptive system – such as that found in schools – the need to nudge/shove everyone into the same track is much more considerable.

The key take home for me from all of this it that we need to be thoughtful about how we communicate with our students. That means not simply setting out what to do, but rather explaining what we’re trying to achieve (in the broad and narrow senses): it doesn’t stop us from recommending techniques to follow, but it does then require us to explain why these might work.

Since I don’t want to paint our school in a completely bad light, they did do this last night when talking about planning revision. As was explained, prioritising topics is a key first step in making a revision timetable: the focus should be on what’s less comfortable or familiar, because that’s where the biggest gains can be, rather than sticking to the stuff you know.

Of course, sometimes even the stuff you know turns out to be not as simple as you might think.

The moral hazard of taking everyone with you

Class yesterday (credit: Urban List)

A common dilemma that I encounter when talking with colleagues about teaching is what to do if students don’t pull their weight.

That might include not doing the reading, not participating, not engaging with opportunities you put out there and all the other ways that students can simply not fit your plans.

As we know all too well from the past couple of years, there are often some very valid reasons about why this is, that have nothing to do with your class, but the effect is the same: they aren’t doing what you think is necessary to succeed and, quite possibly, they are compromising the learning opportunities for their fellow students.

One frequent reaction is to say “well, there are reasons, and we’ve got to be understanding, so I’ve got fallbacks in place.” That might mean access to annotated PowerPoints, or remedial 1-2-1s or whatever.

The difficulty with this is that is creates a strong potential moral hazard: if students know you’ve got their back when they can’t/don’t do the work you ideally intend, then why bother doing that work?

A classic small example of this is asking a class for answers to a question you pose, you getting nothing back, and then telling them the answers you’d like.

Of course, the flipside of this is to say “it’s their call and if they decide not to work for it, then I will just fail them.” No moral hazard, but also no accounting for circumstances that might not be the student’s fault. You might have had a teacher like this in your past, and you probably thought this wasn’t a great approach even then.

So what to do?

Firstly, you need to separate out the general from the specific. If a student has a problematic situation, then you need to have efficient and effective institutional mitigation and tutoring systems: this goes beyond what any one class leader can or should handle. Your institution has specialist support services for precisely this kind of thing, so use them.

But that doesn’t deal with the specific situation of your class, with students who might have other circumstances and students who might not; again, the effect is much the same.

How is where it’s important to think about how you design your class.

At every step, consider what you need students to do and how you can design it so that individual points of failure to do that don’t compromise outcomes any more than they have to.

A couple of examples might help here.

When you flip your class, don’t simply treat it as a case of moving all the knowledge-transmission into a video that can then run directly into an in-person seminar. A student who didn’t, or couldn’t, watch the video will find it very difficult to engage with the discussion, and so lose out on not one, but two sessions.

Instead, see the in-person element as sitting across the flipped content: it recasts ideas and content and opens up different perspectives. If you create your in-person session with lower entry costs and showcase some key ideas, then you not only make it easier for the student to pick up something from that session but you also – because you’re flagging it all the time – give them a good reason to watch the video, to enrich what they have just done.

For that reason, I often run no-prep activities in class. For negotiations, that means a pick-up-and-play scenario, rather than one with lots of pre-reading. For other classes, it might be a small group exercise to identify an example on the relevant topic and then pull together materials from research to present to class. The latter example does various things: it lets students learn from each other and validate the value of their contribution; it promotes cooperation; it means web-enabled devices get used for the class, not chat; it allows me to give instant feedback on data collection, analysis and presentation; and it gives everyone a useful resource for assessment (I take a record of the outputs and share them).

Moreover, these kinds of techniques can help to avoid the “there’s no point now” feeling that students who have missed some of your class often get. The class shouldn’t be a expressway where if you pause for even a bit you get left behind; instead it should be more like one of those giant inflatable play spaces that kids have at their parties, with lots of ways to get back on if you bounce off*.

The choice here isn’t between spoon-feeding or utter indifference, but rather about creating multiple opportunities for students to join in the process of learning, whatever their situation. And yes, that means acknowledging that more engagement is likely to drive more reflection and deeper knowledge that in turn leads to better grades, but it’s not an on-off choice.

* You’ll be relieved to hear I’ve never organised one of these for my kids. Or anyone else’s.

The More Things Change . . .

A follow-up to my post from last month about changing an exam prompt:

I created two exams for this course with the same two-part design. First, answer some multiple choice questions. Second, write additions to a Twine story.

For the second exam, five out of seventeen students wrote in a style that resembled, to varying degrees, that of the story. While this marked a minor improvement over the first exam, students incorrectly applied economic concepts more frequently. The average score for the second exam was lower than that of the first exam.

While my sample size is far too small to determine whether the change was statistically significant, I would like students to do better, and I’m wondering how I might change the exam prompt yet again to facilitate this.

Directory Assistance II

I have occasionally written (examples here and here) about students interpreting assignment and exam prompts in ways that differ from what I intend.

This happened again with the first exam in the undergraduate course that I am teaching this semester. The exam prompt directed students to add to a Twine story. In a class of nineteen students, only one actually wrote text to add to the story. The rest of the students wrote up to three pages that described additions to the story. So here is the prompt for the second exam — changes in bold:

“Play the [link to Twine HTML file in Canvas course shell] game. Write a brief paragraph about one character in the Twine that continues the text of the story and presents the reader with a binary yes/no choice to make about the character. Then write a brief paragraph for each outcome of that choice.  The three paragraphs need to be part of a plot line that reflects one of the following economic development concepts:

[list of concepts students are being tested on]

Write the story, do not describe it.

At the top of your exam, in four sentences or less, 1) identify which of these concepts your plot line demonstrates, and 2) explain how the concept is demonstrated by your plot line.

Your work will be assessed according to the rubric below.”

The second exam is at the end of this week, so I will soon be able to report on whether the revised prompt is more effective.