From formative feedback to assessment outcomes

For reasons best known to others, it’s the end of our first semester here, so that means coursework grades are going back to students.

I was even more interested than usual in this event this time around because something unusual happened with my class: they came to talk with me about their assessment.

I know that might seem mundane, but despite my best efforts my office hours have often resembled one of the remoter oases in a desert: potentially of use, but rarely visited by anyone.

I’d love to tell you what was different this semester, but I genuinely have no idea: I did the things I usually did, so maybe it was a cohort effect. Or not.

In any case, I reckon I sat down for discussions with most of the students and emailed with several others. In those exchanges we typically covered both generic guidance on what was required and specific discussion on students’ plans.

Of course, the big question is whether that helped the students to do better.

At this point, I’ll note that my class had about 35 students and it’s a one-off event so far, so I’m alive to not over-reading the outcomes. Against that, the marking has been confirmed by the second marker.

That said, the main positive outcome was that the bottom half of the class moved up quite markedly. In previous years, I’ve always had a cluster of students who simply didn’t ‘get’ the assessment – a reflective essay – and thus came out with poor marks. This time, I had only a couple of students in that situation, and they appeared (from my records) to have not attended most of the classes, and hadn’t come to talk.

Put differently, the tail was severely trimmed and the large bulk of students secured a decent grade.

What didn’t appear to happen was an overall shift upwards though: the top end remaining where it had been previously.

Again, I’m not sure why this might be. Without another cohort I’m not even sure if my guidance actually did anything for anything.

Quite aside from the specific instance, it does underline for me how little me know about the ways in which our teaching practice does and doesn’t impact on student learning.

In this case, I don’t really know how one could ethically test the impact of formative feedback and support, given the multiple variables at play. If you have an idea, I’d love to hear it.

The Difference Between Good and Bad?

One last post about teaching my redesigned course on development last semester:

Is the ability to follow directions what distinguishes the excellent from the average student?

Writing assignments in my courses require students to synthesize information from a variety of source material into a single, cohesive argument. Exams are no different. My instructions for the final exam included “refer to relevant course readings” and “see the rubric below for guidance on how your work will be evaluated.” The rubric contained the criterion “use of a variety of relevant course readings.”

I assumed that these statements would translate in students’ minds as “my exam grade will suffer tremendously if I don’t reference any of the course readings.” Yet nine of the fifteen students who took the exam did not use any readings, despite having written about them earlier in the semester. Four others only referred to a single reading. Only two students incorporated information from several different readings.  

Maybe I’m wrong, but I don’t think I’m at fault here.



To Quiz or Not to Quiz, Part 3

Some final thoughts on adding in-class quizzes to my course on economic development:

For six of the nine quizzes administered so far, students answered only half of the questions correctly. Given the results of my survey on students’ study habits, I am increasingly convinced that the problem of transfer is contributing to their poor performance. Perhaps I should create a series of real world-based practice exercises for next year’s iteration of this course. These exercises could be an additional connection to the reading assignments.

Even though each quiz has a maximum of four questions, the quiz-taking eats up a significant amount of classroom time. Perhaps I should impose a time limit. If I put the quizzes online for completion outside of class, students will be able to search for correct answers, which defeats my purpose of testing recall to strengthen memory.

The quizzes have helped me identify what students still don’t know. Reviewing questions in class after grading each quiz might have helped students better understand the concepts that they had been tested on. But the final exam that I created for the course (Part 8 below) will allow me to only indirectly infer whether this occurred. Maybe next year I should repeat some of the same questions across multiple quizzes, or introduce summative exams, to get a better idea of whether students are in fact learning what they are being quizzed about.

Links to the original series on redesigning this course:

(Trans)formative Assessment in Teaching

Today I’m attending ISA’s inaugural Innovative Pedagogy Conference in St. Louis.  Victor and I are doing a workshop on using games and simulations to teach political violence, showcasing activities like Survive or Die!, Prisoner’s DilemmaModel Diplomacy, identity exercise, and others.  But I’m most interested in reflecting on the session offered by Matthew Krain and Kent Kille of the College of Wooster on Assessment and Effectiveness in Active Teaching in International Studies. Their framework for formative assessment (that can, in fact, be transformative) is very helpful as an overall approach to teaching.

Continue reading

The price of failure

via GIPHY

After last week’s class discussion about participation, I decided to run an exercise that made it really easy to show the marginal benefit of preparation.

I told students to prepare for a meeting about putting together an agenda for another negotiation, and gave them all specific roles, plus some rules of procedure.

(For those who are looking for Brexit sims, this was a Council working group, putting together an agenda for the Commission to take to the UK to discuss the Political Declaration).

Because it was about formulating an agenda, I hoped that students would see they didn’t need to get too deeply into substantive positions, as long as they could frame the general areas to be covered.

Plus, but giving clear roles and rules, I incentivised everyone to push out their own draft agendas prior to the meeting. In so doing, I hoped they’d see that even a small amount of preparation could have big effects.

Um

Obviously, it didn’t turn out that way.

Continue reading

To Quiz or Not to Quiz, Part 2

As promised in my last post, here are the results of the survey of students’ study habits in my course on economic development. Sixteen of seventeen students in the class completed the survey (after repeated reminders); a good response rate although not statistically-significant sample.

When in class, I . . .

  • write notes on a laptop: 25%.
  • write notes on paper by hand: 75%.
  • do not write notes: 0%.

For quizzes, I usually prepare by (choose all that apply):

  • re-reading assigned articles and book chapters: 18.8%.
  • reviewing notes taken during class: 100%.
  • reviewing notes taken when I read assigned texts the first time: 12.5%.
  • doing nothing: 0%.

While I was pleasantly surprised at the number of students who said they take notes — especially handwritten ones — in class, I told the class that highlighting or underlining is an unproductive study technique. It doesn’t engage the motor and language areas of the brain like writing does. Also, when reviewing prior to a quiz or exam, often one has no memory of why one marked a sentence in a text as important. Whether students will heed my advice, I don’t know.

Continue reading

To Quiz or Not to Quiz

Earlier this year I wrote a series of posts on redesigning my course on economic development. As part of the redesign, I added ten in-class quizzes as a way to reinforce students’ learning of important concepts. From my perspective, the quizzes are simple — a maximum of four questions, many of which can be adequately answered with short phrase or sentence. For example:

Why do the rural poor tend to have more children than the urban rich?

However, students’ scores on the first four quizzes have been quite low. I started wondering if bad study habits were causing the terrible quiz performance. Not being able to read students’ minds or observe their behaviors outside of class, I decided to do something unusual: ask them. I created an anonymous online survey with Google Forms. Here are the questions:

When reading assigned texts, I . . . 

  • write notes about important ideas as I’m reading.
  • highlight, underline, or otherwise mark important ideas as I’m reading.
  • do not write notes or mark up text as I’m reading.

When in class, I . . .

  • write notes on a laptop.
  • write notes on paper by hand.
  • do not write notes.

For quizzes, I usually prepare by (choose all that apply):

  • re-reading assigned articles and book chapters.
  • reviewing notes taken during class.
  • reviewing notes taken when I read assigned texts the first time.
  • doing nothing.

I’ll publish the survey results next week.

Links to the original series on redesigning this course:

Badges and recognizing success

Badges are not exactly a new concept, but like many ideas that come from games, they are still not widely known in the world of higher education and learning.  As many of us know from playing sports or participating in Scouts-style programs as children, badges are physical marks of achievement given to those that show competency at a particular skill or for doing well in a competition. In video games like World of Warcraft, meeting a particular goal is called an ‘achievement’.  Each achievement you earn flashes across the public chat, allowing for recognition, and some of them come with special items or titles in the game.  Defeat the Lich King, and you not only get to bask in the glory of victory, but you earn the title ‘Kingslayer’. While typically meant as a form of visible recognition for an achievement, earning achievements or badges can motivate behavior and can be seen as credentials in their own right.

Motivation, recognition, and credentialing skills–sounds like what we want to do in the classroom, right?

Assessing Student Preparation

This is follow-up to a previous guest post by Joel Moore on the benefits of assessing students’ preparation for class discussion rather than their participation in it. Students are asked to rate themselves on the amount of assigned reading they have completed prior to class, and this forms the basis of their preparation grade.

He has created a web app that simplifies the process. The app is available at https://joeldmoore.com/apps/preparation/.

A video that discusses how to use the app is at https://www.youtube.com/watch?v=SQdK1FwycKw.

Transparent Teaching in Action: Sometimes you aren’t as good as you think you are

In the last few months I’ve been interested in transparent teaching. Basically, transparency is a commitment to all of your students to be transparent about your expectations for a class, particularly when it comes to course assignments. Transparent teaching requires being clear and precise about 3 things in an assignment:

  1. Purpose–explain to the students WHY you are asking them to do this assignment.  Don’t just assume they know how it connects to the material.  Talk to them about the skills and knowledge they will gain and how that is relevant to the course, major, program, or other aspects of their lives.
  2. Tasks–be explicit about the tasks students must follow to complete the assignment.  Sometimes we tell our students to ‘write a 5 page essay’ but don’t realize that many of our students might not know what we mean by an essay–it means different things in different fields, and not all of your students will have taken a polisci class before. Of the three areas, this is where we tend to be the most detailed in our assignments, but even here we can probably be more clear about exactly what we want than we area.
  3. Criteria for Success–the students need to know how their work will be evaluated to determine whether they have successfully completed the assignment or not.  This might mean a rubric, or just a list of what you are going to be looking at.  But one of the most important elements–and the one I messed up on–is it provide examples of successful work.  This might be student work, or something written by a professional working in the field. Want students to write a strong literature review? Show them what a strong one looks like, and talk to them about what makes it strong.  Contrast it with unsuccessful or weak work.  Better yet, give them the examples and the rubric, and have THEM score the work, so they understand how the criteria is applied.

Yes, all of that takes time.  But we owe it to our students to give them every chance to achieve success in the work we assign them.  If we think it is valuable for them to do this work, then we need to give them the detail and time it takes so that success is entirely in their hands.  Plus, doing this has wider benefits.

Research by the Transparency in Learning and Teaching (TILT) team at the University of Nevada in Las Vegas has shown that increasing transparency in our assignments doesn’t just impact their work on those assignments. It leads to big gains in student confidence and sense of belonging in college, particularly for disadvantaged populations who come to college less prepared.  At UNLV, they saw a 20% increase in retention for these students who participated in classes where just two assignments were transformed to become more transparent. I led a project at my university last spring on transparency, and while I’m still evaluating the data, those who participated indicated they found the training sessions and transformation process very useful.  If you want to attend a training on TILT, review their extensive resources, or read the ongoing research, head to their Resources page.

So here I am, someone who has trained on transparent teaching, conducted multiple trainings myself, and doing research in the area.  I’ve also used specifications grading in the past, which is in the transparent teaching wheelhouse.  I’m a pro, right?

Yeah, right.  Every time I think I’ve ‘got’ something when it comes to teaching, I end up learning how much more I have to learn.

I’m teaching a new course this semester called Sex, Marriage, and Violence, and I’m running it largely as a seminar.  Students have to write weekly papers on the readings and pose discussion questions that form the basis for class.  In the syllabus I included a purpose statement for these papers as well as a rubric, and the instructions (I thought) were clear: I wanted a full APA style bibliographic entry for each article or chapter, a 1-2 sentence summary of their main claims or findings, a 500-1000 word analysis of the themes, debates, gaps or issues raised by the readings for the week, and 1-2 discussion questions.

The first papers showed up on Monday, and all but one of them failed to follow this format.  Only a few did the bibliographic entries or summaries; some put the summaries in the main body of the paper (something I explicitly said not to do); only a handful gave discussion questions at the end; and a few wrote essentially annotated bibliographies, with no effort to compare or synthesize the readings.

A couple of students messing up is probably their fault; almost everyone messing up is clearly mine.

The good news is that this is fixable.  Because this is an assignment that starts early and repeats weekly, I was able to catch the problem in week 2.  This is an advantage of using smaller stakes assignments throughout the class–there is less harm to the student if they are completely off base in their efforts, and you have a chance to teach them how to improve. 

To fix this, I recognized a key mistake on my part: while I had provided a rubric, I had not provided an example of successful work.  So I drew up a 2 page outline of a paper.  I didn’t write an entire paper myself.  But the outline showed them the formatting I wanted to see in the paper, such as showing them that I wanted the bibliographic entires and summaries BEFORE the paper proper, and the discussion questions AFTER, not embedded in the text.  And I wrote an example of a thesis statement based on the readings for Week 2 along with a single paragraph showing them how to analyze one of the readings according to that theme.  Drawing up this document took me about an hour, but the students SO appreciated it.  They now had a visual aid to see what I wanted with examples, not just a list of requirements and criteria.  I also allowed them to resubmit their paper by the next class, and throughout the course, will ask students to keep anonymized versions of their work so I will have student-written examples to show future classes.

All of this goes to show that even when you are trying to be transparent, sometimes you might not be able to anticipate what kind of information or models the students need to really understand what you want them to do. When that happens, it is typically OUR fault as instructors, not theirs as students. Now, some students will get it wrong no matter how clear you are, but there are definitely times when we are wrong when we think we are being clear.  I’ve sometimes thought that the ‘provide examples of successful work, and show students why it is successful’ is optional, but on reflection I think this is perhaps the most important thing we can do*. This was an important wake-up call for me that even when I think I’m doing a good job, I can still get it wrong. 

*There are those who question providing examples of successful work lest students simply copy it.  I completely understand the critique, but i think with creativity on our part, this can be overcome. Often I am demonstrating formatting and technique, not content, so as long as you change the content out, there’s little risk of copying.  My go-to example is to make arguments about why cats are better pets than dogs, and I can illustrate the expected structure of an essay, a bibliographic entry or citation, a thesis statement, and using evidence to defend a claim using this very non-political example.

Another point here is that providing a structural template for students who aren’t strong writers is super useful to them.  I’m fine with them imitating my structure–I’m teaching them how to write a strong paper.  I encourage students who have more advanced writing skills to branch away from the suggested structure in my outline/template to find their own style.