What ChatGPT can never do

It’s a belated summer here in Belfast, where I’m attending the UACES conference. As well as a lot else, we’re running a series of L&T panels, including one on assessment.

The prompt for this was the bruhaha over ChatGPT and Why All Assessment Is Impossible (I paraphrase): while we might understand that the problems lie much more with the assessment design than the scary AI, that’s still not really filtered through to all of our colleagues.

Our discussion was really stimulating, both for its breadth and for its reflection on what we are trying to achieve in our assessment – and, by extension, in our teaching.

A concern that was raised was that if AI can produce more encompassing integration of knowledge than any human, almost instantaneously, does this mean the ‘end of thinking’ for our students? Even if the lurid framing might rile, the idea is not to be dismissed.

For me, this prompted the thought that even in an era of all-knowing technology there is still a clear role for us as teaching, instructors and facilitators: to help emancipate our students by giving them the tools to build their agency in the world.

We frequently talk about building self-reflection and criticality in our students. In an age when the challenge is parsing and navigating through too much information, getting students to make informed choices about what to use and how to use it is essential.

And we do that because it gives them a way of standing more firmly in the world, to achieve what they want to achieve.

We don’t (I think) want to produce reproductions of ourselves, but autonomous individuals who can both define their own purpose in life and find ways to realise that purpose.

In this framing of education-as-emancipation, it becomes irrelevant what AI can do, precisely because it’s AI doing it, not the student. As one colleague noted, almost all our students don’t come to university to cheat, but to learn and to develop themselves.

Put differently, if everyone can just turn to ChatGPT, then what gives you the edge is understanding that that technology can and can’t do and understanding how you can use that to your own purposes.

This points to assessment that valorises reflection and critical engagement with knowledge and with arguments, so that the student is able to apply such tools to other situations. It also suggests that we as instructors have to spend more time on assessment that is grounded in individual experience and that recognises there is as much value in being able to articulate your self-awareness as in nominal achievement of a particular task.

A case in point in our discussion was groupwork. Yes, you can mark how well a group functions together, but we know from our own lives that sometimes we have to work with people who aren’t our optimal partners [cough], so there’s as much value in understanding how to cope with and mitigate that scenario as there is in everything being super-positive.

When we can pick up a device and find out a pretty decent amount about any given subject at the drop of a hat, ‘knowing stuff’ isn’t useful in the way it used to (even if we go to lots of pub quizzes), and we need to recognise that in all of our teaching practice.

Indeed, you might argue this is a great example of being self-aware and self-critical that we should be applying to ourselves, even as we apply it to our students.

The odd situation of assessment

It’s May 2023 here in the UK, which means we are having so many public holidays that it feels like Belgium.

As a distraction from celebrating the arrival of someone else that we didn’t select into the position of Head of State, academics have been more focused on the latest stage of strike action by the main union, the UCU.

Right now, that means a Marking and Assessment Boycott (MAB). This means not returning marks/grades to the university, which in turn means there are gaps in calculating degree classifications for completing students.

Rather than get into the mechanics of all this, I’ll instead focus on the new approach that employers have taken to this.

In the past, there would be a relatively small deduction of pay for participating in a MAB: 10-20% was the typical range, reflecting the amount of time that might be involved, plus a bit for the disruption it caused.

This time around, employers seem to be going for a rather different model:

Continue reading “The odd situation of assessment”

How to debrief a simulation

Being able to see what’s happening in your session is probably also important

One of the topics that popped up at various points at TLC and ISA was the question of debriefing.

Everyone who does active learning and sims work agrees it’s important and there was lots of head-nodding whenever it was mentioned. Yes, it’s essential for reconnecting students’ learning within the activity back into their wider understanding and development, so why wouldn’t you agree?

However, at neither conference did anyone really get into what happens in a debrief.

Part of me nearly jumped straight into ‘I’ll write a blog’ mode, but then sensible me rocked up to say ‘maybe check to see if anyone’s already written something about this first’, which is good advice. Well done, sensible me.

And there’s loads of stuff. Here are some highlights:

Continue reading “How to debrief a simulation”

What to do about grading

This guest post comes from Eric K. Leonard, Shenandoah University

I had the pleasure of spending last week at the annual International Studies Association conference in Montreal.  It was a pleasure because I got to hear so many interesting discussions on pedagogy and honestly, ISA wasn’t always that way.  I attended my first ISA meeting in 1997 and the number of pedagogical panels was limited, to say the least.  

But as the years (actually decades but that makes me sound really old) have progressed, ISA has truly embraced the push by members to create space for pedagogical discussions at the annual meeting. I know for me personally, it has made attendance at this conference a joy. 

Amongthe plethora of pedagogical workshops, roundtable discussions, and panels that I attended one topic seemed to keep emerging – the assessment of students or grading. And within that topic, there was a lot of buzz around the idea of “ungrading”.  

I will admit that I generated some of the buzz since I am a vocal advocate of this method.  But in several discussions, this topic was raised without my prompting, albeit not always with the same enthusiasm and positivity that I hold for ungrading.  One thing was shared by most in the room and that was a disdain for grading. In one session, Simon Usherwood even mentioned that his appointment does not require grading!  Frankly, based on the level of jealousy that erupted from his confession, I was surprised he made it out of the room in one piece. So maybe we all can’t stop grading, but what would it mean to start ungrading? 

First, let me state that I am not a fan of the term ‘ungrading’.  As one of my students pointed out, they still get a grade at the end of the semester, so is it really ungrading? I much prefer self-reflective grading or something less clunky (suggestions are welcome). 

Second, ungrading is not one method of assessment.  It entails a lot of different methods that include specification grading, labor-based grading, minimal grading, authentic assessment, and others. The commonality amongst these methods is that students primarily self-assess through a process of self-reflection.  

The method that I employ is often considered the most extreme form of ungrading because as the instructor, I don’t grade any of my students’ work.  Because I teach in a hybrid format that includes a flipped learning model (maybe a conversation for another post) students submit a written assignment every week of the semester.  However, I don’t put a letter grade on those submissions.  

Instead, I provide students feedback to assist in their improvement of future submissions and to push students in terms of how they think about the week’s topic.  So throughout the semester, students never receive a letter grade.  But my university requires that I input a grade at the end of the semester.  

The way in which I ascertain student grades is through a process of self-assessment.  At three different points in the semester, students complete a self-assessment of their learning process.  This self-assessment includes questions about what they learned, what sparked curiosity, and questions about attendance, the number of assignments missed, if any assignments were late, participation level and quality of their comments during our classroom discussions, the effort a student has put forth, and the overall quality of their work, which can be compared to an “Answer Key” that contains examples of exemplary work from students in the course.  After students consider all of these different variables, they are asked to submit a grade and justify that grade.  

I explain to students that in most instances, I either accept that grade or input a higher grade.  And historically, that is the case.  However, I reserve the right to request a conference with a student if I am struggling to see how that student justified their grade.  In my experience, these conferences are rare (less than 1% of students per semester) and typically conclude with the student gaining a better understanding of the self-assessment both in terms of its criteria and its purpose.  In two years of ungrading, I had one student that refused to rethink their grade after a conference and therefore received a grade that I struggled to justify.  

With that brief accounting of how I employ ungrading, let me remark on a few critiques that typically emerge around this form of assessment.  

The first is that students, especially certain demographic groups, will simply overvalue their performance. In other words, students won’t truly self-reflect on their work and they will just provide themselves with a higher grade.  

I had a similar fear when starting this process but I can honestly say this has not been the case.  As mentioned above, only one student was adamant about their earned grade, despite my pointing out statements in their own self-assessment that didn’t seem to square with the perceived grade.  Because my policy is to let students grade themselves, I allowed that student’s perceived grade to stand.  But as mentioned, this has happened ONCE in two years.  Ballpark number, that is 1 in 250 students.  And a quick aside, that student would not have fit the demographic that the literature tells us would contest their grade. 

Second, as one participant stated during a panel at ISA, ungrading is “fake.”  I think what they meant by this statement is that ungrading does not accomplish what it claims to.  

An incomplete list of ungrading claims is that: 

  • Ungrading relieves the stress of grading and promotes greater learning rather than a focus on the grade.  
  • It encourages greater collaboration rather than competition among the students and between the professor and the students.
  •  It recognizes the subjective nature of traditional grading, especially in essay submissions, and allows students to agency in their learning process.

Given the context of the conversation, I believe the “fake” comment was directed at this final claim.  But what claims like this one fails to accept is that when an instructor grades an essay, research paper, or dissertation, there is a subject component to it and the student has no agency.  Thus, that is also fake.  

If we provide students the opportunity to be part of the discussion on their grades, we might get more thoughtful students that are more interested in learning rather than their GPA.  And although they do not have complete agency over the grading process, since I am creating the self-assessment form and providing an understanding of what exemplary work looks like, the level of agency is much higher than in a traditional assessment format.  I don’t understand why people see this as a dichotomous relationship in that students either have agency or they don’t.  The reality is that any increased level of agency for students in the educational process is a good thing and I believe ungrading serves to fulfill that goal.     

One final thought on ungrading.  And this is probably the most important point I can make in this post. 

Ungrading is not for everyone.  

Just like any other pedagogical method, ungrading has to fit with your teaching style, your approach to higher education, and your students.  Thus, this post is not an attempt to advocate that everyone adopt ungrading and to chastise those that refuse. Instead, it is an attempt to shed light on how one professor implements this assessment method in his classroom.  

You will notice that I didn’t discuss in any detail the literature on why we should stop grading or how it benefits the learning process.  This literature exists (in abundance) and engaging it might persuade some to give ungrading a try, but the question that every instructor has to ask themselves is whether they think this could work for their pedagogical approach and their students.  I answered that question with a resounding yes (and I have the student comments to support this claim)!  But that doesn’t mean your answer is the same.    

Notes from a conference III, or: why you hate grading

It’s probably bad to write three posts, one straight after the other, but hey: tough.

We’ve got more grading/assessment-related posts coming from people more on it than me, but a couple of times in the first days of ISA, we’ve had the classic teaching-related panel moment of everyone nodding along to ‘but everyone hates grading, right?’

[Parenthetically, one of those moments led to talk of how good it would be if our jobs didn’t including grading at all, whereupon I noted my job doesn’t include it, to many jealous looks. I’m mostly happy that I have so little grading to do (like a dozen scripts a year little), but part of me does miss it, mainly for the reasons set out here.]

The obvious question that no one really seems to keen to investigate is why we all seem to hate grading.

After all, it’s the main mechanism we have for evaluating student learning and, by extension, whether our teaching is getting the results we intend. It’s more rigorous than a chat at the end of class about learning gain or the impressions we glean from observing debates in the classroom, so why are we so down on it?

Let’s consider the usual suspects.

First up, ‘it’s repetitive’. Sure, if you have a pile of a hundred essays on the same topic, that is super-dull, especially if there’s a lot of overlap in the intellectual sources your students used (your lectures, the core textbook, the readings you suggested): it’s a rehash/mangling of what you’ve taught them and it’s probably not pretty.

So don’t make that your assessment. Give them choice, make them responsible for finding their own sources, get them to bring something more personal and individual.

My old negotiation course asked students for their only assessment point to tell me what they had learnt from the course: I was emphatic that it could be absolutely anything, as long as it was grounded in what had happened in class and was connected out to whatever literature they had found useful. Result was such a rich variety, every time I ran that course.

Second up, ‘it takes such a long time.’ Partly that’s about the time-elongating/mind-numbing of reading the same stuff a bunch of times that we just discussed, but partly it’s our demands of the process. Often, we normalise the essay of X pages or Y thousand words as proof of learning, even as we rool our eyes at ‘all that text’.

So go for concision, not length. You know it’s harder to produce a good short text than a good long one: the demand to focus much more tightly on what’s important, to lose the blah, is something we find tricky (to judge by the number of ropey abstracts you and I have read). Set a much shorter length piece, or ask for a 2-page policy brief, or a poster, or an infographic, or a 3 minute video.

Of course, shorter assessment outputs doesn’t automatically mean quicker grading, but it helps. As does trying things like videoing your feedback, or asking students to make their own evaluations of their or their peers’ work.

Finally, ‘students find it boring/unpleasant (and so do I)‘. Find me the student who loves to do assessments and I’ll just start wondering what happened to them to make it come to this. I get that everyone thinks it’s an imposition and chore.

So involve your students in finding assessments that engage them more/turn them off less. Amanda’s breakfast exercise for research methods is my go-to here: super-fun, super-engaging allowing them to really capture their experiences in an assessment.

If assessment is about cementing the bond of teacher and learner, then why wouldn’t you want to bring them in to find a mutually-satisfactory assessment regime?

Unless you’re working under a highly prescriptive institutional system, you hold the power to make your assessment and grading less boring. So use it.

The robots are coming! And they’re… writing essays!!

It’s a truism that no academic is actually interested in assessment. Sure, there are certainly academics who find assessment stimulating and and engaging as a topic, but none of us has ever met one.

Except you have: me.

I never really understood the antipathy towards assessing: maybe it’s a carryover from being a student, where being tested felt, in very large part, like being punished. And I’m not going to pretend that I really liked sitting down to go through scripts.

One big exception to that was the reflective piece that my negotiation students wrote about their work, exploring and expanding on what they had taken from the course and contextualising it in the wider literature. Every one was a fascinating insight into my students’ heads, in a way that 4,000 words on ‘IR is overly fixated on power’ never is.

Any way. The only point were colleagues do seem to get more interested is when they have a problem with their assessment.

Right now, that problem is ChatGPT.

If you’ve somehow missed this one, this is an AI text generator, capable of creating extensive – and seemingly well-written – responses to minimal prompts. Including to things like essay titles.

To say that colleagues have been concerned is very much an understatement.

Both online and in-person, I’ve seen colleagues describe the system as the death knell of the essay format in assessment. Text is both of a standard that it could satisfy criteria to get a passing grade and that isn’t going to trigger anti-plagiarism software (since it’s not cut-and-paste, but organically created).

For all the rumours that this latter software will become upgraded to pick up on such AI-generated text, the feeling is deeply pessimistic.

My own view is perhaps more measured, mainly because with all the examples I’ve seen I haven’t felt the output is that impressive, especially for any one looking to bypass they way to the kind of higher grade that so many students seek out.

As others have noted, the quality and rigour of such texts isn’t up to much, which means both that it’s possible to pick up on AI generation (even if evidencing a plagiarism case is still a massive pain in the neck) and that students get a dubious amount of return (in grade terms).

But the bigger point is that text is only part of how assessment works.

The choice of questions you ask and the requirements you impose on students also matter massively.

Take that negotiation class I mentioned. Because I knew all the students, saw all they did in class and debriefing them extensively at the time about their learning, I had a very clear idea of what might be in their reflective pieces.

So if someone tried to write about stuff they hadn’t done (and a couple tried), I knew and could mark accordingly.

More generally, this all should be making us think more carefully about what assessment is for. And part of that is acknowledging that the very large majority of students don’t want to cheat on their education: sure it’s less effort (they think), but it screws when they are out in the world, trying to use skills or understanding they don’t actually have.

So I leave you with this example of how we can get students to engage with these challenges. It’s not a whole solution, but it is a recognition that blind panic or utter despair aren’t helpful responses.

We’ll be coming back to this several more times, no doubt, but I’d love to hear your thoughts on the matter.

Ungrading

A short one today, to encourage you to read this thread from C Thi Ngiuyen on how he’s challenging students’ understanding of grading:

Suffice to say here that his ideas resonate a lot with my own, but he’s in a position to do more about it with his class. For those of you who are bound to your institutional requirements on grading and assessment, this is still a really useful discussion to have, with both students and colleagues.

Assessing large groups

Among the many upsides of working with relatively small groups of students for most of my career has been that I’ve not been driven primary by volume management concerns. I could look across at my colleagues in Law or Business and laugh at the thought of having to juggle several hundreds of exam scripts at once.

(One time, a Business lecturer proudly told us how he’d marked 400 exams in three days, only for it to become clear it was a multiple-choice paper, with answers scanned in, which raised questions about why it had taken so long.)

But in the spirit of cooperation, a recent tweet about the need to treat assessment as an integral part of our teaching activity prompted this response from a Legal colleague:

This is an issue for many of us at some point: the big compulsory course where innovation feels like a luxury.

So what do to?

Sylvia’s dilemma is three-fold: assessment needs to a) serve learning objectives, b) minimise opportunities for cheating, and c) be practical to turn around with reasonable speed. We’ve not had the chance to speak about her specific situation, so what follows should be read more generically.

My personal view is that we always have to place learning objectives first in assessment: do we test for the things that we consider it essential that the students should have learnt?

In any course or module that covers a variety of elements: substantive knowledge; research skills; presentational and interpersonal skills; and more general aspects of critical thinking and building confidence. That breadth is important, because it underlines that ‘knowing facts’ isn’t the be-all and end-all here: even for us in academia, we probably make as much use of the skills and competences we gained from our study as we do the knowledge (and we’re at the high end of a spectrum of knowledge use).

Sylvia mentions vivas as a nominally ideal form of assessment, possibly because it’s interactive and personal and offers lots of opportunities to test how far a student can work with what they know. Having sat through vivas for a course of 100 students, I could point up some issues, but the analysis still holds: here’s something that better serves the learning objectives.

So are there other ways to get that same benefit without the big time implications of a viva system?

Two strategies suggest themselves, if we’re treating final written exams as an unsatisfactory option: different formats and collaborative working.

Asking students to produce posters, infographics or podcasts not only opens up different ways of presenting material, but also requires considerable distillation of substantive knowledge into key points, which in turn stimulates more critical engagement. Yes, students will be unfamiliar with the practicalities, but this can be covered with some preparatory sessions, and it develops presentational skills that might be otherwise neglected.

If you want to stick to text, then asking for shorter written pieces – policy briefs, submission to a court – can also keep the focus on distillation, plus give experience in formats they might encounter in their work (unlike a long-form essay).

And all of these options could be used with collaborative approaches too. Learning to work together is a valuable skill [he writes from his shed], so why not test for that? Group projects can be marked for the group as a whole, plus with individual marking for short reflective pieces on what each person contributed and got from it.

Of course, free-riding is an issue, and some disciplines might encounter accreditation barriers on collaborative assessment, but the problems need not be any greater than for final exams.

The right answer will vary from case to case: your capacities; the nature of your course; your institution’s attitude; the willingness of your students to buy into it. But these discussions are still worth having. Just because things have ‘always been like this’, doesn’t mean they should continue like this, especially if it’s not working for you or your students.

If you have more ideas on this, or what to chat about your assessment, drop me a line and we’ll talk.

The interrelation between attendance and writing assignment in a PBL course

This guest post comes from Patrick Bijsmans (Maastricht University) and Arjan Schakel (University of Bergen)

In one of his recent contributions to this blog, Chad asks why students should attend class. In his experience

[C]lass attendance and academic performance are positively correlated for the undergraduate population that I teach. But I can’t say that the former causes the latter given all of the confounding variables.

The question whether attendance matters often pops up, reflected in blog posts, such as those by Chad and by Patrick’s colleague Merijn Chamon, and in recent research articles on the appropriateness of mandatory attendance and on student drop-out. In our own research we present strong evidence that attendance in a Problem-Based Learning (PBL) environment matters, also for the best students, and that attending or not attending class also has an influence on whether international classroom exchanges benefit student learning.

Last year we reported on an accidental experiment in one of Patrick’s courses that allowed us to compare the impact of attendance and the submissions of tasks in online and on-campus groups in Maastricht University’s Bachelor in European Studies. We observed that that attendance appeared to matter more for the on-campus students, whereas handing in tasks was important for the online students.

This year the same course was fully taught on-campus again, although students were allowed to join online when they displayed symptoms of or had tested positive for Covid-19 (this ad-hoc online participation was, unfortunately, not tracked). We did the same research again and there are some notable conclusions to be drawn.

In the first-year BA course that we looked at, students learn how to write a research proposal (see here). The course is set up as a PBL course, so it does not come as a big surprise that attendance once again significantly impacted students’ chances of passing the course.

Continue reading “The interrelation between attendance and writing assignment in a PBL course”

Learning about teaching

Blossoming

Last night found me at our kids’ school, for a talk on revising. Aside from being a reminder of how quickly people decide that facemasks aren’t prudent any more, it brought home some lessons about the way we construct teaching for others.

The talk was primarily a run-through of what will be happening after Easter with exams, plus subject-specific sections on useful resources and good revision practice. Its content was much as you imagine, and as familiar to me (as a teacher) as it was to my daughter (who’s now on her third time of hearing it all in as many weeks).

So what’s worth mentioning here, on a site devoted to university education, where we don’t (usually) draw parents into it all?

The teachers here have clearly had some training on revision, including some useful models of ‘how to revise’, which they brought to the table. But what was missing (for me at least) was an unpacking of how revision fits into the broader process of learning.

Back at the start of the year, we got a welcome talk about what the next cycle for our daughter would be (the two years up to her first major external exams). In that was lots of stuff, but not so much about how keeping materials and notes would be a key part of ‘revising’, in the sense that they discussed last night. Revision implies vision a first time, and all the revision techniques set out in the current talk require a baseload of substantive knowledge and understanding to be able to produce the materials for effective assessment performances.

Put the other way around, if you’d not done the work until now, having six weeks until the tests to revise as the school would like you to is not a viable proposition.

And this is where this all matters for you (and me). Assessment (and by extension, revision) is too often treated as a disparate element of the educational experience; something tagged on the end, just because we have to.

Instead, assessment is an integral part of learning and should be handled as such, a logical extension to what happens in a class and in a student’s broader package of work through a programme.

This disconnect was evident in a couple of other places too, last night.

One of the teachers asked that students didn’t come to them, asking for ‘help with something vague’, but rather with a precise and focused query: ‘I have tried to do this past paper question on topic X and I can’t seem to make sense of it, despite several tries’, seemed to be the preferred line.

Now, as a teacher, I appreciate that more precision means more scope to get into the nuts and bolts with a student, but I also appreciate that the bigger problem is students not coming to ask for help at all. If I were a student who was struggling, being told I now needed to come with a precise inquiry strikes me as more daunting.

Here the issue is one of assumption-making about student engagement and buy-in to the programme of study. Even the most wonderful teaching set-up does not guarantee that engagement and we always have to be alert to those students that haven’t found their place within it.

That’s best treated not as the fault of the student, or the teacher, but of the specific instance. In a university setting we have more discretion to change and adapt that instance to accommodate individuals on a different path, but in a much more prescriptive system – such as that found in schools – the need to nudge/shove everyone into the same track is much more considerable.

The key take home for me from all of this it that we need to be thoughtful about how we communicate with our students. That means not simply setting out what to do, but rather explaining what we’re trying to achieve (in the broad and narrow senses): it doesn’t stop us from recommending techniques to follow, but it does then require us to explain why these might work.

Since I don’t want to paint our school in a completely bad light, they did do this last night when talking about planning revision. As was explained, prioritising topics is a key first step in making a revision timetable: the focus should be on what’s less comfortable or familiar, because that’s where the biggest gains can be, rather than sticking to the stuff you know.

Of course, sometimes even the stuff you know turns out to be not as simple as you might think.