I’m still enough of a kid to be excited to see the place I work at mentioned in the news, especially if it’s in an outlet my mum might see.
Of course, it’d be better if the context of this particular mention were different, but I guess you can’t have it all.
This all comes off the back of the on-going debate in government about grade inflation.
I wrote about all this last summer, and I’m not sure I’ve gotten much further in my thinking about this, except to note the shift in framing to combating ‘artificial’ grade inflation.
While this might seem to start to take account of the other factors at play, what it singularly doesn’t do is set out a means of calculating this in practice.
Obviously, there are changes in student characteristics that have a direct bearing and these are relatively simple to capture: socio-economic status; entry grades; progressive performance in each year of study.
However, there are also obviously changes in the teaching environment: staffing changes; changes in pedagogic approach; changing curricula (we’ve made our final year dissertation optional thus year, for example); changing provision of learning resources outside the degree programme, at the library or in welfare; changes in programme regulations.
Nikita Minin of Masaryk University is motivated by a goal we can all appreciate: ensuring that his students achieve the learning outcomes of his course. In his case, the course is a graduate seminar on theories of IR and energy security and the learning outcomes include improving student skills in critical thinking and writing. He noticed that students in his class did not seem to really improve on these skills during the class, and introduced three teaching interventions in an attempt to fix this.
First, Minin provided more intense instruction on the writing assignments at the start of the course, providing a grading rubric and examples of successful student work. Second, he gave students audio rather than written feedback on their papers. Finally, using a sequential assessment system, the instructor gave formative feedback first and grades much later in the course. Minin assessed the impact of these three interventions, comparing course sections with and without them, and concluded that the first two interventions achieved the objective of improving student achievement of the learning outcomes.
The interventions described in the chapter are in line with current thinking regarding in-course assessment. While Minin does not use the language of transparent teaching, his first intervention falls exactly in line with the Transparency in Teaching and Learning Project’s (TILT)approach. Transparency calls on instructors to openly communicate about the purpose of an assignment, the tasks they are to complete, and the criteria for success, and Minin does exactly that in this first intervention. Given the data so far on the TILT project, it is not surprising that Minin saw some success by taking this approach. Likewise, now-ubiquitous learning management systems allow for giving feedback in multiple platforms, including audio and video. For years now, advocates for audio-based feedback claim that this can be a more effective tool than written feedback. Minin’s observations therefore, also fit nicely in line with existing work.
Where the chapter falls short, then, is not in the design of its interventions, but in the claims made based on the available data. The sample sizes are tiny, with just five students receiving the interventions. With final grades used as the primary dependent variable, it is difficult to tease out the independent impact of each of the three changes. Using final grades is also an issue when the experimenter is also the person who assigns grades, as it is more difficult to avoid bias than when more objective or blind items are used. Lang’s (2016) bookSmall Teaching: Everyday Lessons from the Science of Learningtells us that engaging in self-reflection is itself an intervention, and Minin’s use of minute-paper style self-reflections to assess the impact of feedback, while itself an interesting and potentially useful idea, mean that a fourth intervention was used in the course. While I do not doubt Minin’s observations that his interventions had a positive impact, as they are backed by existing research, the evidence in the chapter does not strongly advance our confidence in those findings.
However, I have never been one to dismiss good teaching ideas simply because of a lack of strong evidence from a particular instructor. Minin highlights a crucial concern—that we should never assume that our courses are teaching what we intend them to teach, and that ‘time and effort’ do not necessarily achieve the desired results, even for graduate students. Reflecting on this, seeking out innovative solutions, and then assessing the impact is a process we should all be following, and Minin sets a great example.
My first-year module this semester has been a real training ground for me. Not only am I going all-in on flipping, but I’m also trialing the new assessment software that the University is thinking of using.
By extension, that also means it’s a training ground for my students, something that I’ve been very open about with them.
The flipping seems to be working and I’ll be writing up my thoughts on that later in the semester, but having coming through the first use of the software I need to make some decisions now.
In part, my situation arises from wanting to push how we used the software past a conventional approach. Not only did students submit a literature review to it, but they then had to review someone else’s using the system, all in aid of a final piece of self-reflection (which we’re marking now).
Using the marking function is a bit more involved than just submitting work and a couple of people did get a bit lost on that. But the bigger problem was that not everyone submitted work.
In the good old days (i.e. last year and before) we did all this in-class, so it was much simpler to cover (the exceptionally few) missing pieces. However, because we’d pre-selected peer reviewers, we ended up with some students having nothing to review and others not getting their work reviewed.
That’s a failing on my part: next time, I’d leave allocation until after the first submission was in, so everyone who submitted got allocated and reviewed.
But that’s next time. What about now?
Already, I’ve indicated to everyone that not getting peer feedback won’t count against them in marking, but a couple of students have felt that absent such comments they’re not in a position to complete the self-reflection.
To that, I’ve had to underline that it’s self-reflection, so peer feedback was only ever one component of that: indeed, the whole purpose of the somewhat-convoluted exercise is to get students becoming more independent and critical about their learning.
All that said, peer review was added in here to help prompt everyone to think more about what they’ve done and what they could do.
As we sit down to mark, the question will be much we can, and should, take the circumstances into account. Until we’ve seen the full range of work, that’s going to be a tricky call to make.
However, it all highlights an important point in such situations: do we have fall-backs?
Trying new things is inherently risky – that’s why many colleagues stick with what they know – but with some risk management, that need not be a barrier to moving practice forward.
Annoying through our situation here is, it’s not fatally-compromising to the endeavour: we know who’s affected and how; they’re still able to submit work; and the assessment is relatively small in the overall scheme of things.
Yes, we’ll be using the system again for the final exam, but without the aspects that have proved problematic. Indeed, the exam has already been trialled elsewhere in the University, so that’s well-understood.
So, on balance, I feel comfortable that we can manage the situation and implement the necessary changes next time around to remove the problems identified.
Which is, of course, a big part of the reason for trying it out in the first place.
For reasons best known to others, it’s the end of our first
semester here, so that means coursework grades are going back to students.
I was even more interested than usual in this event this
time around because something unusual happened with my class: they came to talk
with me about their assessment.
I know that might seem mundane, but despite my best efforts
my office hours have often resembled one of the remoter oases in a desert:
potentially of use, but rarely visited by anyone.
I’d love to tell you what was different this semester, but I
genuinely have no idea: I did the things I usually did, so maybe it was a
cohort effect. Or not.
In any case, I reckon I sat down for discussions with most
of the students and emailed with several others. In those exchanges we
typically covered both generic guidance on what was required and specific
discussion on students’ plans.
Of course, the big question is whether that helped the
students to do better.
At this point, I’ll note that my class had about 35 students
and it’s a one-off event so far, so I’m alive to not over-reading the outcomes.
Against that, the marking has been confirmed by the second marker.
That said, the main positive outcome was that the bottom
half of the class moved up quite markedly. In previous years, I’ve always had a
cluster of students who simply didn’t ‘get’ the assessment – a reflective essay
– and thus came out with poor marks. This time, I had only a couple of students
in that situation, and they appeared (from my records) to have not attended
most of the classes, and hadn’t come to talk.
Put differently, the tail was severely trimmed and the large
bulk of students secured a decent grade.
What didn’t appear to happen was an overall shift upwards
though: the top end remaining where it had been previously.
Again, I’m not sure why this might be. Without another
cohort I’m not even sure if my guidance actually did anything for anything.
Quite aside from the specific instance, it does underline for
me how little me know about the ways in which our teaching practice does and
doesn’t impact on student learning.
In this case, I don’t really know how one could ethically
test the impact of formative feedback and support, given the multiple variables
at play. If you have an idea, I’d love to hear it.
One last post about teaching my redesigned course on development last semester:
Is the ability to follow directions what distinguishes the excellent from the average student?
Writing assignments in my courses require students to synthesize information from a variety of source material into a single, cohesive argument. Exams are no different. My instructions for the final exam included “refer to relevant course readings” and “see the rubric below for guidance on how your work will be evaluated.” The rubric contained the criterion “use of a variety of relevant course readings.”
I assumed that these statements would translate in students’ minds as “my exam grade will suffer tremendously if I don’t reference any of the course readings.” Yet nine of the fifteen students who took the exam did not use any readings, despite having written about them earlier in the semester. Four others only referred to a single reading. Only two students incorporated information from several different readings.
Maybe I’m wrong, but I don’t think I’m at fault here.
Some final thoughts on adding in-class quizzes to my course on economic development:
For six of the nine quizzes administered so far, students answered only half of the questions correctly. Given the results of my survey on students’ study habits, I am increasingly convinced that the problem of transfer is contributing to their poor performance. Perhaps I should create a series of real world-based practice exercises for next year’s iteration of this course. These exercises could be an additional connection to the reading assignments.
Even though each quiz has a maximum of four questions, the quiz-taking eats up a significant amount of classroom time. Perhaps I should impose a time limit. If I put the quizzes online for completion outside of class, students will be able to search for correct answers, which defeats my purpose of testing recall to strengthen memory.
The quizzes have helped me identify what students still don’t know. Reviewing questions in class after grading each quiz might have helped students better understand the concepts that they had been tested on. But the final exam that I created for the course (Part 8 below) will allow me to only indirectly infer whether this occurred. Maybe next year I should repeat some of the same questions across multiple quizzes, or introduce summative exams, to get a better idea of whether students are in fact learning what they are being quizzed about.
Links to the original series on redesigning this course:
Today I’m attending ISA’s inaugural Innovative Pedagogy Conference in St. Louis. Victor and I are doing a workshop on using games and simulations to teach political violence, showcasing activities like Survive or Die!, Prisoner’s Dilemma, Model Diplomacy, identity exercise, and others. But I’m most interested in reflecting on the session offered by Matthew Krain and Kent Kille of the College of Wooster on Assessment and Effectiveness in Active Teaching in International Studies. Their framework for formative assessment (that can, in fact, be transformative) is very helpful as an overall approach to teaching.
After last week’s class discussion about participation, I decided to run an exercise that made it really easy to show the marginal benefit of preparation.
I told students to prepare for a meeting about putting together an agenda for another negotiation, and gave them all specific roles, plus some rules of procedure.
(For those who are looking for Brexit sims, this was a Council working group, putting together an agenda for the Commission to take to the UK to discuss the Political Declaration).
Because it was about formulating an agenda, I hoped that students would see they didn’t need to get too deeply into substantive positions, as long as they could frame the general areas to be covered.
Plus, but giving clear roles and rules, I incentivised everyone to push out their own draft agendas prior to the meeting. In so doing, I hoped they’d see that even a small amount of preparation could have big effects.
As promised in my last post, here are the results of the survey of students’ study habits in my course on economic development. Sixteen of seventeen students in the class completed the survey (after repeated reminders); a good response rate although not statistically-significant sample.
When in class, I . . .
write notes on a laptop: 25%.
write notes on paper by hand: 75%.
do not write notes: 0%.
For quizzes, I usually prepare by (choose all that apply):
re-reading assigned articles and book chapters: 18.8%.
reviewing notes taken during class: 100%.
reviewing notes taken when I read assigned texts the first time: 12.5%.
doing nothing: 0%.
While I was pleasantly surprised at the number of students who said they take notes — especially handwritten ones — in class, I told the class that highlighting or underlining is an unproductive study technique. It doesn’t engage the motor and language areas of the brain like writing does. Also, when reviewing prior to a quiz or exam, often one has no memory of why one marked a sentence in a text as important. Whether students will heed my advice, I don’t know.
Earlier this year I wrote a series of posts on redesigning my course on economic development. As part of the redesign, I added ten in-class quizzes as a way to reinforce students’ learning of important concepts. From my perspective, the quizzes are simple — a maximum of four questions, many of which can be adequately answered with short phrase or sentence. For example:
Why do the rural poor tend to have more children than the urban rich?
However, students’ scores on the first four quizzes have been quite low. I started wondering if bad study habits were causing the terrible quiz performance. Not being able to read students’ minds or observe their behaviors outside of class, I decided to do something unusual: ask them. I created an anonymous online survey with Google Forms. Here are the questions:
When reading assigned texts, I . . .
write notes about important ideas as I’m reading.
highlight, underline, or otherwise mark important ideas as I’m reading.
do not write notes or mark up text as I’m reading.
When in class, I . . .
write notes on a laptop.
write notes on paper by hand.
do not write notes.
For quizzes, I usually prepare by (choose all that apply):
re-reading assigned articles and book chapters.
reviewing notes taken during class.
reviewing notes taken when I read assigned texts the first time.
I’ll publish the survey results next week.
Links to the original series on redesigning this course: