Ungrading

A short one today, to encourage you to read this thread from C Thi Ngiuyen on how he’s challenging students’ understanding of grading:

Suffice to say here that his ideas resonate a lot with my own, but he’s in a position to do more about it with his class. For those of you who are bound to your institutional requirements on grading and assessment, this is still a really useful discussion to have, with both students and colleagues.

Assessing large groups

Among the many upsides of working with relatively small groups of students for most of my career has been that I’ve not been driven primary by volume management concerns. I could look across at my colleagues in Law or Business and laugh at the thought of having to juggle several hundreds of exam scripts at once.

(One time, a Business lecturer proudly told us how he’d marked 400 exams in three days, only for it to become clear it was a multiple-choice paper, with answers scanned in, which raised questions about why it had taken so long.)

But in the spirit of cooperation, a recent tweet about the need to treat assessment as an integral part of our teaching activity prompted this response from a Legal colleague:

This is an issue for many of us at some point: the big compulsory course where innovation feels like a luxury.

So what do to?

Sylvia’s dilemma is three-fold: assessment needs to a) serve learning objectives, b) minimise opportunities for cheating, and c) be practical to turn around with reasonable speed. We’ve not had the chance to speak about her specific situation, so what follows should be read more generically.

My personal view is that we always have to place learning objectives first in assessment: do we test for the things that we consider it essential that the students should have learnt?

In any course or module that covers a variety of elements: substantive knowledge; research skills; presentational and interpersonal skills; and more general aspects of critical thinking and building confidence. That breadth is important, because it underlines that ‘knowing facts’ isn’t the be-all and end-all here: even for us in academia, we probably make as much use of the skills and competences we gained from our study as we do the knowledge (and we’re at the high end of a spectrum of knowledge use).

Sylvia mentions vivas as a nominally ideal form of assessment, possibly because it’s interactive and personal and offers lots of opportunities to test how far a student can work with what they know. Having sat through vivas for a course of 100 students, I could point up some issues, but the analysis still holds: here’s something that better serves the learning objectives.

So are there other ways to get that same benefit without the big time implications of a viva system?

Two strategies suggest themselves, if we’re treating final written exams as an unsatisfactory option: different formats and collaborative working.

Asking students to produce posters, infographics or podcasts not only opens up different ways of presenting material, but also requires considerable distillation of substantive knowledge into key points, which in turn stimulates more critical engagement. Yes, students will be unfamiliar with the practicalities, but this can be covered with some preparatory sessions, and it develops presentational skills that might be otherwise neglected.

If you want to stick to text, then asking for shorter written pieces – policy briefs, submission to a court – can also keep the focus on distillation, plus give experience in formats they might encounter in their work (unlike a long-form essay).

And all of these options could be used with collaborative approaches too. Learning to work together is a valuable skill [he writes from his shed], so why not test for that? Group projects can be marked for the group as a whole, plus with individual marking for short reflective pieces on what each person contributed and got from it.

Of course, free-riding is an issue, and some disciplines might encounter accreditation barriers on collaborative assessment, but the problems need not be any greater than for final exams.

The right answer will vary from case to case: your capacities; the nature of your course; your institution’s attitude; the willingness of your students to buy into it. But these discussions are still worth having. Just because things have ‘always been like this’, doesn’t mean they should continue like this, especially if it’s not working for you or your students.

If you have more ideas on this, or what to chat about your assessment, drop me a line and we’ll talk.

The interrelation between attendance and writing assignment in a PBL course

This guest post comes from Patrick Bijsmans (Maastricht University) and Arjan Schakel (University of Bergen)

In one of his recent contributions to this blog, Chad asks why students should attend class. In his experience

[C]lass attendance and academic performance are positively correlated for the undergraduate population that I teach. But I can’t say that the former causes the latter given all of the confounding variables.

The question whether attendance matters often pops up, reflected in blog posts, such as those by Chad and by Patrick’s colleague Merijn Chamon, and in recent research articles on the appropriateness of mandatory attendance and on student drop-out. In our own research we present strong evidence that attendance in a Problem-Based Learning (PBL) environment matters, also for the best students, and that attending or not attending class also has an influence on whether international classroom exchanges benefit student learning.

Last year we reported on an accidental experiment in one of Patrick’s courses that allowed us to compare the impact of attendance and the submissions of tasks in online and on-campus groups in Maastricht University’s Bachelor in European Studies. We observed that that attendance appeared to matter more for the on-campus students, whereas handing in tasks was important for the online students.

This year the same course was fully taught on-campus again, although students were allowed to join online when they displayed symptoms of or had tested positive for Covid-19 (this ad-hoc online participation was, unfortunately, not tracked). We did the same research again and there are some notable conclusions to be drawn.

In the first-year BA course that we looked at, students learn how to write a research proposal (see here). The course is set up as a PBL course, so it does not come as a big surprise that attendance once again significantly impacted students’ chances of passing the course.

Continue reading “The interrelation between attendance and writing assignment in a PBL course”

Learning about teaching

Blossoming

Last night found me at our kids’ school, for a talk on revising. Aside from being a reminder of how quickly people decide that facemasks aren’t prudent any more, it brought home some lessons about the way we construct teaching for others.

The talk was primarily a run-through of what will be happening after Easter with exams, plus subject-specific sections on useful resources and good revision practice. Its content was much as you imagine, and as familiar to me (as a teacher) as it was to my daughter (who’s now on her third time of hearing it all in as many weeks).

So what’s worth mentioning here, on a site devoted to university education, where we don’t (usually) draw parents into it all?

The teachers here have clearly had some training on revision, including some useful models of ‘how to revise’, which they brought to the table. But what was missing (for me at least) was an unpacking of how revision fits into the broader process of learning.

Back at the start of the year, we got a welcome talk about what the next cycle for our daughter would be (the two years up to her first major external exams). In that was lots of stuff, but not so much about how keeping materials and notes would be a key part of ‘revising’, in the sense that they discussed last night. Revision implies vision a first time, and all the revision techniques set out in the current talk require a baseload of substantive knowledge and understanding to be able to produce the materials for effective assessment performances.

Put the other way around, if you’d not done the work until now, having six weeks until the tests to revise as the school would like you to is not a viable proposition.

And this is where this all matters for you (and me). Assessment (and by extension, revision) is too often treated as a disparate element of the educational experience; something tagged on the end, just because we have to.

Instead, assessment is an integral part of learning and should be handled as such, a logical extension to what happens in a class and in a student’s broader package of work through a programme.

This disconnect was evident in a couple of other places too, last night.

One of the teachers asked that students didn’t come to them, asking for ‘help with something vague’, but rather with a precise and focused query: ‘I have tried to do this past paper question on topic X and I can’t seem to make sense of it, despite several tries’, seemed to be the preferred line.

Now, as a teacher, I appreciate that more precision means more scope to get into the nuts and bolts with a student, but I also appreciate that the bigger problem is students not coming to ask for help at all. If I were a student who was struggling, being told I now needed to come with a precise inquiry strikes me as more daunting.

Here the issue is one of assumption-making about student engagement and buy-in to the programme of study. Even the most wonderful teaching set-up does not guarantee that engagement and we always have to be alert to those students that haven’t found their place within it.

That’s best treated not as the fault of the student, or the teacher, but of the specific instance. In a university setting we have more discretion to change and adapt that instance to accommodate individuals on a different path, but in a much more prescriptive system – such as that found in schools – the need to nudge/shove everyone into the same track is much more considerable.

The key take home for me from all of this it that we need to be thoughtful about how we communicate with our students. That means not simply setting out what to do, but rather explaining what we’re trying to achieve (in the broad and narrow senses): it doesn’t stop us from recommending techniques to follow, but it does then require us to explain why these might work.

Since I don’t want to paint our school in a completely bad light, they did do this last night when talking about planning revision. As was explained, prioritising topics is a key first step in making a revision timetable: the focus should be on what’s less comfortable or familiar, because that’s where the biggest gains can be, rather than sticking to the stuff you know.

Of course, sometimes even the stuff you know turns out to be not as simple as you might think.

The moral hazard of taking everyone with you

Class yesterday (credit: Urban List)

A common dilemma that I encounter when talking with colleagues about teaching is what to do if students don’t pull their weight.

That might include not doing the reading, not participating, not engaging with opportunities you put out there and all the other ways that students can simply not fit your plans.

As we know all too well from the past couple of years, there are often some very valid reasons about why this is, that have nothing to do with your class, but the effect is the same: they aren’t doing what you think is necessary to succeed and, quite possibly, they are compromising the learning opportunities for their fellow students.

One frequent reaction is to say “well, there are reasons, and we’ve got to be understanding, so I’ve got fallbacks in place.” That might mean access to annotated PowerPoints, or remedial 1-2-1s or whatever.

The difficulty with this is that is creates a strong potential moral hazard: if students know you’ve got their back when they can’t/don’t do the work you ideally intend, then why bother doing that work?

A classic small example of this is asking a class for answers to a question you pose, you getting nothing back, and then telling them the answers you’d like.

Of course, the flipside of this is to say “it’s their call and if they decide not to work for it, then I will just fail them.” No moral hazard, but also no accounting for circumstances that might not be the student’s fault. You might have had a teacher like this in your past, and you probably thought this wasn’t a great approach even then.

So what to do?

Firstly, you need to separate out the general from the specific. If a student has a problematic situation, then you need to have efficient and effective institutional mitigation and tutoring systems: this goes beyond what any one class leader can or should handle. Your institution has specialist support services for precisely this kind of thing, so use them.

But that doesn’t deal with the specific situation of your class, with students who might have other circumstances and students who might not; again, the effect is much the same.

How is where it’s important to think about how you design your class.

At every step, consider what you need students to do and how you can design it so that individual points of failure to do that don’t compromise outcomes any more than they have to.

A couple of examples might help here.

When you flip your class, don’t simply treat it as a case of moving all the knowledge-transmission into a video that can then run directly into an in-person seminar. A student who didn’t, or couldn’t, watch the video will find it very difficult to engage with the discussion, and so lose out on not one, but two sessions.

Instead, see the in-person element as sitting across the flipped content: it recasts ideas and content and opens up different perspectives. If you create your in-person session with lower entry costs and showcase some key ideas, then you not only make it easier for the student to pick up something from that session but you also – because you’re flagging it all the time – give them a good reason to watch the video, to enrich what they have just done.

For that reason, I often run no-prep activities in class. For negotiations, that means a pick-up-and-play scenario, rather than one with lots of pre-reading. For other classes, it might be a small group exercise to identify an example on the relevant topic and then pull together materials from research to present to class. The latter example does various things: it lets students learn from each other and validate the value of their contribution; it promotes cooperation; it means web-enabled devices get used for the class, not chat; it allows me to give instant feedback on data collection, analysis and presentation; and it gives everyone a useful resource for assessment (I take a record of the outputs and share them).

Moreover, these kinds of techniques can help to avoid the “there’s no point now” feeling that students who have missed some of your class often get. The class shouldn’t be a expressway where if you pause for even a bit you get left behind; instead it should be more like one of those giant inflatable play spaces that kids have at their parties, with lots of ways to get back on if you bounce off*.

The choice here isn’t between spoon-feeding or utter indifference, but rather about creating multiple opportunities for students to join in the process of learning, whatever their situation. And yes, that means acknowledging that more engagement is likely to drive more reflection and deeper knowledge that in turn leads to better grades, but it’s not an on-off choice.

* You’ll be relieved to hear I’ve never organised one of these for my kids. Or anyone else’s.

The More Things Change . . .

A follow-up to my post from last month about changing an exam prompt:

I created two exams for this course with the same two-part design. First, answer some multiple choice questions. Second, write additions to a Twine story.

For the second exam, five out of seventeen students wrote in a style that resembled, to varying degrees, that of the story. While this marked a minor improvement over the first exam, students incorrectly applied economic concepts more frequently. The average score for the second exam was lower than that of the first exam.

While my sample size is far too small to determine whether the change was statistically significant, I would like students to do better, and I’m wondering how I might change the exam prompt yet again to facilitate this.

Directory Assistance II

I have occasionally written (examples here and here) about students interpreting assignment and exam prompts in ways that differ from what I intend.

This happened again with the first exam in the undergraduate course that I am teaching this semester. The exam prompt directed students to add to a Twine story. In a class of nineteen students, only one actually wrote text to add to the story. The rest of the students wrote up to three pages that described additions to the story. So here is the prompt for the second exam — changes in bold:

“Play the [link to Twine HTML file in Canvas course shell] game. Write a brief paragraph about one character in the Twine that continues the text of the story and presents the reader with a binary yes/no choice to make about the character. Then write a brief paragraph for each outcome of that choice.  The three paragraphs need to be part of a plot line that reflects one of the following economic development concepts:

[list of concepts students are being tested on]

Write the story, do not describe it.

At the top of your exam, in four sentences or less, 1) identify which of these concepts your plot line demonstrates, and 2) explain how the concept is demonstrated by your plot line.

Your work will be assessed according to the rubric below.”

The second exam is at the end of this week, so I will soon be able to report on whether the revised prompt is more effective.

Collecting Data From Students

As previously discussed, this semester I am attempting to research whether metacognitive exercises improve students’ learning — as measured by exam scores. My class completed the first survey and exam. A few initial impressions about the data and the process of collecting it:

Eighteen students completed the pre-exam survey a total of twenty-seven times. Two students submitted responses three times each. This demonstrates the importance of requiring that students include some kind of identifying information when they complete the survey, so that duplicate submissions can be removed from the data set.

I suspect the survey data are skewed because of above average effect or subject bias. By coding the responses from 1 to 5, with 1 being “never” and 5 being “always,” the highest possible sum score from the ten survey questions is 50. The average for this survey was 40. I doubt students actually engaged in the study strategies referenced in the survey as frequently as they said they did.

The average total score on the exams five multiple choice questions was 7.7 out of 10. Given the small sample and the nature of the data, a statistical analysis that compares these scores against survey responses isn’t meaningful, but I did run a correlation in Excel, which resulted in a very non-impressive r of -0.12.

The exams in this course are extremely low stakes — the first and second exams are worth 25 points each, and the final exam is worth only 40 points, out of more than 1,000 points available from all graded items. That might have affected how diligent students were in studying for the exam.

Given the small size of the class and the usual host of possible confounding variables, I can already guess that I won’t be able to identify a relationship between the metacognition surveys and students’ exam performance. Repeatedly asking students about their study techniques might help them learn more, but I’m not going to be able to demonstrate it.

Failing to succeed

One thing that has been really good about being part of ALPS has been the community around it.

For example, this week’s post is inspired by my former colleague and general force of nature, Maxine David, who pushed out this thread the other day (click to read it all):

Essentially, Maxine’s asking the same question that I think we’ve all asked at some point: what are we trying to achieve in our classes?

As you’ll see from the responses to the thread, I started to sketch out a position, but I’d like to expand on it here some more.

Amanda and Nina have long championed failure in the classroom as a valuable learning experience for students. Their argument – which I also hold to – is that hitting nominal targets is good, but not a complete education: not hitting them encourages students to reflect more on the process of learning (and application) that they’ve undertaken. Think of it as being analogous to playing a game, where not hitting the (rather different) target makes you go back and try again, with the thought of why it didn’t work before in your mind.

This model requires us to acknowledge that learning has multiple targets.

Yes, we want students to know stuff and know how to do stuff (which we can catch with summative assessments), but we also want students to know how to know all this. Becoming a reflexive learner and a critical thinker is a core skill for building capacity to learn throughout the rest of one’s life and it’s a skill that has no easy metric, nor any obvious threshold.

And thresholds were my first thought when I read Maxine’s thread.

When we assess, we typically look for evidence of meeting some threshold: does the student demonstrate that they know enough about X or enough about how to do Y? Those thresholds are present in our grading and those institutional matrices that benchmark us all to common standards.

[Cough]

Maxine rightly points out that we cannot really ever separate out the formative and summative elements of assessment: if we genuinely value the development of reflexive learning, then we absolutely shouldn’t be trying to separate them out in the first place.

But this position is vanishingly rare in academia these days. Yes, I tell my doctoral students that a good viva should see every singly person coming out of the room having learnt something, but even that’s not a given.

Easy as it would be to blame the pressures of QA culture and metrification for all this, we also have to recognise that we often don’t create opportunities within our own classes. Even if we aren’t allowed to make adjustments for support received (as Maxine suggests), we should still be trying to instil a culture of collaboration, reflection and development among our students and between them and us.

In so doing we might start to reclaim some of that learning opportunity that will serve everyone in the class well, wherever they are and whatever they do.

UPDATE:

You might have seen that England is going through some very pointed discussions about racism, following the European football championships. This tweet from one of the national team players exactly captures the point:

Formative Assessment: Abort, Retry, Fail?

Two can play this game

Something of a response to Simon’s June 1 post on transitioning from pedagogical theory to teaching practice: he wrote, in part, “assessment is always formative and should be always linked to the feedback and adaptation process.” In theory, I agree. In practice, while I can lead students to feedback, I am still unable to make them read it.

As I’ve written before, the Canvas LMS has a “student viewed” time stamp feature that shows whether a student looks at my feedback on an assignment — my comments and a tabular rubric with cells that I’ve highlighted — after I have graded it. Generally, though, given the lack of time stamps, many students simply ignore this information. An example, with data: my annual spring semester course on comparative politics. In 2018 and 2019, I taught this course in the physical classroom. In 2020, the latter half of the course was online because of the coronavirus pandemic. In 2021, the course was delivered online for the entire semester. For each iteration, I tallied the number of students who looked at the first three, the third to last, and the second to last reading responses after I graded them. Results are below. N is number of students in the class; not every student in a class completed every assignment. The eyeball columns indicate the how many students viewed an assignment after I had graded it; the eyeball with a slash is the opposite.

While I can understand students not bothering to revisit assignments that they earned full marks on, I don’t understand why students who earn less than full marks frequently ignore information that would allow them to do better in the future. Anyone have an explanation?