Writing as Assessment

To illustrate the dilemma I presented in my last post: the possible devolution of the final exam for one of my courses.

My Fall 2018 exam was an attempt at an authentic writing exercise, but students had to choose one of two positions and use course readings. This meant I supplied the available arguments and evidence, and the exam was actually an assessment of rhetorical skill. Students didn’t demonstrate an ability to use concepts that I thought were crucial for the arguments they had developed.

For the final exam in Fall 2019, I ended up giving students a choice of arguments — “basis for aid policy to Egypt” and “plan for Louisiana’s future” — and I added this to the instructions for the exam:

Apply relevant concepts like discounting the future, moral hazard, etc.

Students still had to select one of two predetermined positions regardless of the argument chosen, and again I specified the pool of evidence they could draw from. And students still didn’t demonstrate knowledge of concepts listed in the exam’s instructions.

What next? I could have a final exam that asks students to, for example, “make an evidence-based determination of whether moral hazard exists in a location affected by climate change.” But this type of exam prompt might introduce even more problems.

The Joy of Documenting a Job Done

Not knowing whether one has actually helped students learn is one of the most frustrating aspects of teaching. Assuming an absence of megalomania or the Dunning-Kruger effect, indications that we’ve made a difference are typically quite vague and ambiguous. So I was pleasantly surprised — as in, “hey, maybe students really did benefit from this” — by the results of a knowledge probe that I launched at the beginning and end of the semester in my course on economic development and environmental politics.

The knowledge probe was an ungraded quiz that asked questions about a few basic economic concepts, administered through Google Forms in the first and last weeks of the semester. Results, showing percentage of respondents answering correctly, are below.

Pre
N = 21
Post
N = 16
% Change
Poverty Trap5210092
Diminishing Returns to Capital527544
Skill Matching5881,660
Common Pool Resource Problem488169
Moral Hazard38100163

Obviously this wasn’t the perfect tool. Sample sizes are too small for statistical significance. And a slightly different proportion of students reported previously taking a university economics course on the pre-test than on the post-test. But the numbers at minimum suggest that students learned something over the semester, which gave me a sense of satisfaction that I otherwise wouldn’t have.

Petty Officer Usherwood?

It’s nearly Christmas, if anyone’s short of ideas

Yesterday I had about a third of my students turn up for class. Possibly that was related to the deadline for an essay due for me later that day – certainly the number who turned up was about the same as the number who’d already submitted the work.

Since I’ve known that this was going to be an awkward timing since the start of semester, back in early October, I’d left some of the session open, so I could be flexible about what to do, including not asking for any specific prep beyond the general reading.

In the event, I spent a block of the class talking about assessment. Unsurprisingly, since they’d already submitted, none of the students who turned up wanted to talk about the essay, but they did want to talk about the exam, which’ll be after the Christmas break.

So we discussed how that would work (we’re doing a seen-paper format, so they get it a week beforehand) and what I was looking for.

So what’s the problem?

Well, the people who turned up yesterday are the ones who most likely didn’t need the discussion, either because they’d have worked out the salient points already, or because they’d have asked. Indeed, the student who asked about the exam some weeks ago was there.

The issue is for those who didn’t turn up, the ones still working on their essay a couple of hours ahead of the deadline, the ones will the poor attendance throughout the semester.

This is a classic of the Matthew Principle: those that have, get more. And it’s not really helped by me being a bit petty-minded.

I could have waited until next week’s final class to discuss the exam – and probably someone who wasn’t there yesterday will ask about it – but I have also spent two months trying to reinforce the message that the rational choice for a student who’s finding it hard going is to come to class, because that’s the best place to get the essentials together, and to get questions answered.

Partly, this is about incentives. For my other class, on negotiation, I have great attendance, mainly because the classwork is very active and because the assessment is about what you’ve done in class. In this case, the work is more mixed and it’s not directly linked.

Maybe I need to be thinking about whether I can change that, in a way that works for the subject matter.

But maybe I also need to think more about how much this is a case of taking horses to water: where do me responsibilities lie and where do they end?

Maybe one for class discussion next week.

Exam Study Skills Advice

If you’re like me, you are finding that more students need help with basic study skills than previously. You might also find it difficult to explain learning strategies that for many academics have been automatic behaviors since elementary school. Loleen Berdahl of the University of Saskatchewan has created a handy screen-capture video about studying effectively for final exams, available here, just in time for the end of the semester in the USA and Canada. Feel free to share it with your students.

Helping students do better in essays: what are the options of the seminar tutor?

This guest post is part of a series linked to the publication of G. Pleschova & A. Simon (eds.) Learning to teach in central Europe: Reflections from early career researchers. This post comes from Liudmila Mikalayeva.

Daniela Jaklová Střihavková tells her story of a highly motivated seminar tutor in a first-year course on social work. The problem she sets to solve is a mismatch between the grader’s expectations and the students’ performance: the average grade for exam essays is between 16 and 18 points in 2016-2017, which is far from the maximum 30 points. What can the tutor do to fill this gap between the expectations and the performance?

The setup of the course will be familiar to many young instructors: while a senior professor gives weekly content-focused lectures and grades students’ work at the end of the course, a junior tutor is in charge of interactive seminars where students discuss and apply knowledge. The examination consists mainly of two essays checking students’ understanding of the nature of social work and of the social worker’s role, formulated in proper academic language. The essays are challenging for students, since they receive little training in writing them in the course itself.

Jaklová Střihavková very reasonably suggests improving the course through constructive alignment: course design should guarantee the fit between teaching goals, teaching methods and assignments. So if a major teaching goal is to enable students to define the role of the social worker in a concrete case, they should receive guidance on how to do it, have an opportunity to practice it and receive feedback before the exam.

Practically, the author introduced two additional seminar sessions where students practiced a task similar to that of the exam essay. Combining individual and group work, she provided space for the students to confront the complex task and reflect on their own performance. While she cannot prove that the essays received better grades because of these changes, both the grader and the students were happier with the learning outcome.

The effort by the seminar tutor to bridge the gap between the expectations of the grader and the actual students’ work was however only partly successful. Even after the additional seminars, students continued to feel unsure what the grader expected from them and the grader was still unhappy with how they used disciplinary vocabulary. I see three issues explaining the persistence of the gap.

A relatively minor point is that oral exercises may not be effective enough for supporting students’ success in written tasks. A much more important drawback, underlined by the author herself, is the absence of clear and explicit criteria for grading: the professor would need to make an effort detailing the requirements. And, most significantly, the course structure as such is at the core of the problem: the person grading students’ work is not directly in touch with the students and is senior enough to have forgotten how challenging it is for undergraduate students to understand and use academic jargon and navigate the often-implicit expectations to their work.

Jaklová Střihavková is right to point that to improve learning outcomes students need space to reflect on the curriculum, but young and senior instructors should become more reflective as well. Clarifying expectations, aligning content, teaching approaches and assignments, communicating among themselves and with the students is key and cannot be replaced by teaching experience alone. Students as well as instructors will benefit from it.

Unprepared students? Try an Online Quiz

This guest post is part of a series linked to the publication of G. Pleschova & A. Simon (eds.) Learning to teach in central Europe: Reflections from early career researchers. This post comes from Alistair Jones

We’ve all had the problem of students turning up to class without adequate preparation. To gain the full benefits of any classroom discussion or debate, student preparation is essential.  Yet there are too many other distractions, resulting in unprepared students attending class.  Natália Gachallová explains how she got the students to prepare for their classes, the ensuing benefits, and embeds the explanation in the existing literature.

Preparation for an introduction to Latin anatomical nomenclature, you might think, would be essential. The obvious incentive is the required mark of 75% to pass the first part of the course. Yet experience suggested this was not the case. Thus Gachallová decided to innovate her teaching. She introduced fortnightly summative online mini quizzes. These would reinforce prior learning as well as provide a useful revision tool. There was also a reward component, where students gained extra marks for the final exam based on the average score in the quizzes.

Such innovation can be time consuming, especially noting the length and volume of classes that are undertaken. In this case, there is a cohort of over 130 students! Gachallová does not mention how much time was used in preparing these quizzes, especially in comparison to what preparation was undertaken previously. Follow-up questions were used in class to stimulate debate – an example would be interesting to see.

An online student survey was utilised to measure the effectiveness of this approach, which produced remarkable findings. Around 85% of respondents claimed to find the in-class quizzes beneficial. Conversely, some respondents complained about the quizzes being too challenging, and voiced concerns over spelling mistakes leading to marks being dropped.

The benefits are visible in the grades of students. Both the average mark and the overall pass rates improved. The exception is in the tail of the class, where pass rates at the third attempt of sitting the final exam were almost unchanged.

Yet Gachallová takes into consideration the extra marks gained by students from the online quizzes. Her findings showed most students did not need the extra marks from the quizzes.  Most of those who passed the exam would have passed without the extra marks. A very small number of students failed the exam despite gaining the extra marks from the online quizzes. The reward component was meaningful for about 5% of all students.

The key message from this chapter is simple. If students can engage with the learning materials throughout the academic year, they are far more likely to pass. Online quizzes appear to motivate the students to engage with the class preparation. Formative assessments AND formative feedback can increase student success in the summative assessments.

Some of you may consider a Latin Medical Terminology course to be rather niche. It might be that online quizzes are not deemed appropriate for other subjects. Yet that is to miss the bigger picture. There is a need to engage with the students; to encourage them to take responsibility for their learning. One way is through the use of technology. The fact a small incentive was added in relation to the marks may be inconsequential – which is something for future exploration. 

If students are not going to engage with classroom preparation, why hold the class? If students merely parrot what has been said in class, has any learning actually happened?  Consider the six levels of knowledge in Bloom’s Taxonomy (1956): to have knowledge, comprehension, application, analysis, synthesis and evaluation. If using technology and incentives to innovate our teaching can help our students to a deeper level of knowledge, then it is worth the experimentation.


 [GP1]Simon, can you please insert chapter link here?

Wheel Reinvention Avoidance

A short one today, as I’m in the middle of the UACES conference in Lisbon, trying out my novel panel formats and I’m worrying about the tech side of it all: more on that next week.

On Sunday, we had our regular pre-conference L&T workshop, which was structured to let us discuss in small groups what and how we teach. The initial idea had been to focus this on different substantial specialisations, but it was evident that we all ended up working more general issues and questions.

One of the most useful thoughts in this was the notion that this kind of sharing is really helpful, because it lets us avoid having to do everything from scratch.

Rather than building from the ground up, the exchange of ideas can mean both the avoidance of things that don’t work and the opening-up of new vistas.

To take the latter, I got a couple of really interesting ideas from the discussion, that I’m now looking to see if I can implement with my students.

The first is assessing student’s lecture notes.

This could be as simple as just checking they’ve made such notes, or – as this is where I can see real potential – you could ask for those notes to be augmented with further reading/noting-making, or framed with some elaboration of how different elements hang together, or coupled to a reflective piece of writing.

The second is getting students to present each other’s work.

This forces students to think/act a lot more in making their work understandable enough for someone else to talk it through, and to appreciate how others interpret your work: did they stress what I wanted them to in their presentation?

Neither is such a radical idea, but they contain the core of some new things to explore.

Does the Question Determine the Answer?

Regular readers of this blog know that I sometimes ponder the clarity of my assignment and exam prompts (some past posts on this subject are here, here, and here). Students sometimes don’t hit what, in my mind, the question targets, so I revise in the hopes of creating a prompt that is more transparent. But I don’t want prompts to be answerable with a Jeopardy-like regurgitation of facts. I want students to exert some cognitive effort to figure out how to apply concepts that are relevant to the question at hand.

Usually this situation occurs with my undergraduates, but I’m noticing it more frequently with master’s degree students. A recent example is an assignment from my graduate-level introduction to comparative politics course:

Continue reading

What Do Grades Mean?

What do grades actually mean? I began pondering this question while designing a course for the fall semester. Theoretically a grade indicates the amount of knowledge or skill that a student possesses. But really? Those of us working in the USA are quite familiar with grade inflation. A final grade of C today probably doesn’t indicate the same level of knowledge or skill proficiency as the C from fifty years ago. There is also the persistent problem of knowing whether our assessment tools are measuring the types of learning that we think they are/want them to. And it is probably safe to assume that, both in and out of the classroom, there is a lot of learning happening but we just aren’t interested in trying to measure it. The situation gets even more complex given that — again, in the USA — a “learning activity” often won’t function as intended if students believe that it has no discernible effect on their course grades.

I structure my syllabi so that the sum total of points available from all assessed work is greater than what it needed for any particular final grade. For example, a student might need to accumulate at least 950 points over the semester for an A, but there could be 1,040 points available. I do this to deliberately create wiggle room for students — with so many assignments, students don’t need to get perfect scores on, or complete, all of them. While this leads to higher grades in my courses than if I graded strictly on a bell curve, I want to give students plenty of opportunities to practice, fail, and improve. And I firmly believe that sloppy writing indicates slopping thinking, while good writing indicates the converse. So in reality what I’m doing with most of my assignments is evaluating the writing abilities of my students.

This system often produces a bimodal grade distribution that is skewed to the right. Expend a lot of effort and demonstrate a certain level of proficiency, and you will get a grade somewhere between an A and a B-. Choose not to expend the effort, or consistently demonstrate an inability to perform at a minimum level, and you will get a D or an F. I’m comfortable with this result, in part because I know from the cognitive science research on learning that repeated exposure and frequent testing builds long term memory.

This leads me to the reason for doubting that grades my courses mean the same thing as they do in courses where the only assessment is done through mid-term and final exams composed of multiple-choice questions. Yes, the proportion of A’s in the latter might be lower than in the former, but I bet on average my students are retaining more. At least I like to think that’s the case. There is no way for me to be sure.

More Changes to a Course on Development, Part 3

The final exam for this course last year asked each student to write an economic rationale in support of one of two policy options, using information from course readings as evidence. Generally students students did not do well on the exam, mainly because they did not discuss applicable concepts like moral hazard and discounting the future. These concepts were found in several course readings and discussed in class. While I didn’t explicitly mention these concepts in the exam prompt, the benefits of including them in the rationale should have been obvious given course content.

Now I’m thinking of a question like this for the final exam:

What has a greater influence on economic development in Egypt: law (institutions) or geography (luck)? Why?

In your answer, reference the items below and relevant course readings listed in the syllabus:

The downside here is that I’m giving up an authentic writing exercise in favor of (I assume) even more transparent alignment with course objectives.