Can we make our students into superforecasters?

Poster_of_Alexander_Crystal_SeerAs long-standing readers might recall, I read. A lot. But I don’t tend to talk/write about my reading much, either because it’s not immediately relevant or because the rest of my life is so rich with incident. Probably the former, then.

However, I’ve just ploughed through Philip Tetlock & Dan Gardner’s “Superforecasting“, which does offer some interesting perspectives that might be tied to our work in the classroom.

Tetlock will be known to you for the meme that most predictions are no better than chance in their accuracy. As he points out, while that might be true in the aggregate, it’s not true for ever forecaster, and this book (which follows up the Expert Political Judgment that set out the (rigorous) basis for the meme) explores those people who have demonstrated a consistent and measurable ability to outperform both the metaphorical ‘crowd’, but also other mechanisms, such as forecasting markets.

Tetlock’s core point is that such ability is not innate, but learnt and learnable. And that’s what interests me here.

Over the past couple of weeks, I’ve been teaching students about negotiation theory, before I drop them off into the metaphorical deep end of actual negotiation. One of the core skills I’ve been trying to stress to them is the need for preparation, which forms the bedrock of good practice.

However, while it’s easy to say “be prepared”, it’s often hard to know how to prepare, especially if the situation is a complex one. And I can see utility in Tetlock’s approach as a way of improving the chances of having as good an oversight of the preparatory phase as possible.

In practical terms, that means doing things like breaking down intractable problems into tractable sub-problems, looking at the situation from as many different perspectives as possible and engaging in constant checking of your practice and biases.All the kinds of things we might be encouraging students to do as active learners, in fact.

And perhaps that’s the point. Learning is a memetic process, where we draw analogies across from one place to another, hopefully casting light on the way. Tetlock’s not interested in this book about effective actors, although he does mention several examples of how his core ideas get operationalised, but about our ability to forecast events. I’m less interested in that, but can see how I could use it to improve students’ performance elsewhere.

Tetlock talks about the rise of the ‘mission command‘ model in the military, the notion that what matters is the underlying intent, which local commanders can then find the most appropriate way to achieve it. This replacement of the strict hierarchy of command-and-control with something much more flexible and adaptable is not so different from the shift towards active learning. We set out learning objectives and students find their own path towards them.

I wouldn’t want to stretch this too far, but it makes the point that we should be casting our nets as widely as possible when developing our practice, because the analogies can take us to places that might never have considered before.

A Quick Exercise on Confirmation Bias and Hypothesis Testing

This neat exercise featured on the New York Times takes a few seconds to play and includes a neat set of examples of how confirmation bias impacts government policy and corporate America.

Basically, you are presented with 3 numbers in a sequence, and asked to guess the rule that governs the sequence. You can enter in any 3 numbers you like, and the system will tell you whether or not your sequence follows the rule or not. When you are ready to guess, you enter it in but you receive no second chances. Apparently 78% of people make a guess without getting a single ‘no’–and most get the rule wrong.

The example in the NY Times is ‘2, 4, 8’. A number of possible rules could come to mind–must contain multiples of 2, or even numbers, or that the number doubles the one before it. The actually rule in this case is even simpler: the number must be larger than the one before it, meaning that ‘4, 8, 16’ works, but so does ‘1, 10, 3593’.

They don’t mention it in the article, but this exercise can adapted to teach hypothesis testing. Used in class, you can put the sequence on the board and have students suggest other sequences, which you then judge as either following or not following the rule. They have to use this information to come up with the right answer.

If this sounds familiar, it may be because one of the very first entries on this blog was about the board game Zendo, which does precisely this, only with physical pieces rather than numbers. I still use Zendo on day 1 of my methods class, and find it a really useful tool for teaching a variety of methodological skills. This numerical version is a great, easy activity to pull out for a quick fix on helping students with their logical thinking.

Fear and Loathing of Broccoli

Fear and LoathingI very rarely incorporate feature films into my courses — Dr. Strangelove has been one of the few exceptions — but anyone who is teaching political psychology should take a look at Inside Out from Pixar. The film, for which the psychologists Paul Ekman and Dacher Keltner functioned as consultants, is a visual representation an 11-year old girl’s mind. Emotions take the center stage, especially in regards to memory formation and retrieval, but imagination, attention, and other processes also play into the film’s plot. More detail on the science incorporated into Inside Out is here.

Confirmation Games

KatnissThe New York Times recently published an interactive illustration of confirmation bias: guess the rule obeyed by a sequence of three numbers. I won’t go into detail about the game other than to say that it’s wonderfully simple and I definitely fell into the mind trap. Some implications for politics and business are presented after players submit their answers and this can provide a launch point for class discussion.

The puzzle nicely complements the Zendo game in which players create hypotheses about an arrangement of blocks — by demonstrating the cognitive biases that affect much of our decision making.

The Best Laid Plans

Cat and MouseA while back I wrote a series of posts on reworking my first-year seminar. My assumption was that this fall’s version would meet three days a week, as happened in the course’s initial iteration. I recently learned that instead it will meet on Tuesdays and Thursdays. Given that much of the course involves student-to-student interaction in the classroom, the new schedule necessitated further changes. To start, I dropped the book that I had originally fit into the last third of the semester, and with it plans for a class-wide Twine project. The course now looks like this:

  • Team-based Twines on the book An Ordinary Man (Rwanda).
  • Simulation exercises on the first four cases in Chasing Chaos (Rwanda, Sudan, Sri Lanka, Sierra Leone).
  • Team-based Twines on the last case in Chasing Chaos (Haiti).

Since this is a course for incoming college students, I added The New Science of Learning and some other meta-cognitive content on skills for academic success. This means that students will have on average three writing assignments on readings per week even though the class only meets twice a week, which I think this is a good thing. Students won’t be able to forget about the course between Thursdays and Tuesdays.

As I discussed in my informal assessment back in January, I had a problematic formulation for the briefing memo that prepared students for each Chasing Chaos simulation. I’ve rewritten the assignment instructions accordingly, and created a new sample memo for students to use as a guide. The effort that I’m putting into the design of this course reflects something about how college works that I’ll discuss in more detail in my next two posts.

Book Review: The New Science of Learning

New Science of LearningWe’ve talked previously about Classroom Assessment Techniques by Thomas Angelo and K. Patricia Cross (here, here, and here), so I thought I’d post a review of another book:

Terry Doyle and Todd Zakrajsek, The New Science of Learning: How to Learn in Harmony with Your Brain, Stylus Publishing, 2013.

The New Science of Learning is a very concise and easy-to-read advice guide for undergraduates that is based on the findings of cognitive science research. I’ll be using it this fall in my first-year seminar. My hope is that it will help students, many of whom are not that well prepared for college, improve their academic performance. Here is one of the book’s authors speaking at Quinnipiac University, from the perspective of how to teach more effectively.

I’ve created these writing assignments that correspond to the chapters of the book:

  • Of the different practices that help people learn more effectively, which is the one that you currently use the least frequently? What would you need to change in your life so that you used it more frequently?
  • Think about the last three nights. How well did you sleep on each of these nights? What changes would enable you to sleep better? How can you implement these changes?
  • Thinks about the last three days. At what times were you physically active and for how long? How did your levels of mental alertness change during the day? Do you notice any pattern between physical activity and alertness?
  • Do you write notes by hand in your college courses? Do you annotate text that is assigned in these courses? Why? Given the benefits on learning of note-taking and annotating reading assignments, how well will you perform academically this semester? 
  • Describe an assignment in one of your courses this semester that reflects the pattern recognition principle of similarity/difference, proximity, figure-ground, or cause/effect. What is the assignment and how does it reflect the principle? What will be the effect on your understanding or memory of the material?
  • Name an activity in which you use either distributed practice or elaboration. What is a specific change that you can make in your daily behavior to better incorporate either one into your college experience?
  • What is your approach to failure? Do you embrace the possibility of it or try to avoid it at all costs? When you fail, what is your reaction? Based on your answers to these questions, do you have a growth mindset or a fixed mindset toward learning? Why?
  • Do you engage in task shifting? When? What is a specific change that you can make in your daily behavior to reduce task shifting?

Fail Safe Not

Fail SafeThis is the fifth semester that I have used the quality of failure essay as an end-of-semester exercise in meta-cognitive reflection, after Amanda* first posted about the concept almost three years ago.

Even though the essay is only two to three pages, I make it worth the equivalent of half of a letter grade because it forces students to acknowledge that they are responsible for their own learning. Or so I thought. 

Out of my forty-five students this semester, sixteen failed to submit the assignment on time or at all. This occurred despite an eight-day window in which students could upload their work to the course website and my continuing effort to prevent them from engaging in learned helplessness. Those who did not meet the deadline ranged from freshmen to seniors and most of them had performed at a mediocre (or worse) level throughout the semester. Some approached me after the final to ask if they could turn in this and other unsubmitted assignments to salvage their grades (no). Conversely and as usual, the best students submitted their essays the earliest. 

To avoid the stream of “Uh, I forgot” emails next semester, I’m creating a “failure to learn from failure” assignment, worth half as much as the quality of failure essay. This second essay will be only for those students who do not meet the deadline of the first one, and I’ll set the window of availability for forty-eight hours. Students will have to specify why they failed to take advantage of the first essay in addition to following the same instructions on the essay’s content.

Although my perceptions might be getting clouded by age, I believe that my experience this semester is part of an increasingly stark bimodal distribution among students at my employer. Some come in already possessing the motivation and skills necessary to succeed in college. A nearly equal portion arrive assuming that they will never be held accountable for their willed ignorance. Unfortunately some in the latter category apparently never discover in four years of college that this assumption is false.

*recently tenured and promoted, evidence that good things can happen to good people.

Presentation Fail

More on this semester’s comparative politics course:

Blind MonkeyAfter doing the in-class exercise on how to produce a concise thesis statement, I created this template for students to use when writing essays in which they apply theory to historical events. We shall see if students take advantage of it on the next essay assignment.

Student presentations have also been problematic. My instructions for this task have been:

Your team’s presentation needs to discuss which theoretical perspective (rational actor, structure, or culture) best matches the readings for your theme for this geographic region. Include specific examples from the readings to support your argument.

Teams create their presentations after their members have individually written essays that accomplish the same function, a deliberate sequence on my part. However, the presentations have been terribly organized — no clear thesis statement and few to no examples drawn from readings that actually support whatever argument students think they are communicating.

It’s possible that my directions are still too broad and that students need more step-by-step instructions; if so, the easy solution is to modify the template that I created for essays and require that teams use it when designing presentations. 

I don’t really want to do this. I prefer students to be creative in their approach to solving problems and to take responsibility for their learning. To continuously break tasks down into smaller pieces and decrease the need for effort or ingenuity risks turning students into box-checking monkeys. Yet without enough structure it is unlikely that an assignment will serve its intended purpose.

This tension reflects the difficulty in overcoming the problem of transfer. My assignments — which in this course include fourteen one-page responses to readings, five multi-page essays, and five presentations — represent multiple opportunities for each student to develop a single skill, the effective communication of an argument. Yet students don’t see this. They are blind to the possibility that a technique that they have learned to use in one context can be successfully applied in another one. After fifteen years as a professor, I still am trying to figure out how to move students from needing a list of steps to follow to being able to recognize that they already have the tools needed to figure things out for themselves. 

When Grade Inflation Isn’t

Norman Mailer Report Card

Courtesy of Charles Gleek at Games Without Frontiers,  I became aware of an interesting discussion of grade distributions at TPRS Q & A:

What grades should kids get? Notes on Evaluation for the Mathematically Challenged.

It’s a bit long but worth reading. The author, Chris Stolz, points out that perceptions about “proper” grade distributions are sometimes based on ignorance of basic statistical principles. Classes frequently are not statistically representative — they contain too few students, and students choose to enroll in them for non-random reasons. Assessment instruments often produce a ceiling effect that masks evidence of improvement in students who come into a course already possessing a high level of proficiency.

The end result can be a class composed of students who are either predisposed to do well in the course (possibly the main reason they enrolled in it to begin with) or who learn enough over the course of a semester to earn high marks on summative assessments. This reduces variation in the grade distribution and skews the curve to the right — instead of a normal distribution from F to A, with most students getting a C, the majority of the class ends up with A’s and B’s. A person who does not understand statistics assumes this happened because of grade inflation.

Several years ago I abandoned grading students against a normal distribution curve for these reasons. I also became much less concerned with testing students’ ability to reproduce factual information on “objective” exams because I knew that the vast majority of what they regurgitated would never move into their long-term memories. I did not (and still don’t) believe that their lives would be fundamentally altered for the worse if they failed to remember that the Turks captured Constantinople in 1453 or that realist IR theory derives in large part from the writings of Thucydides, Hobbes, and Machiavelli. 

I thought that students would benefit more from multiple opportunities to demonstrate how well they could apply concepts in novel ways and effectively communicate their findings. How does this look in reality? Below is my grading system for a course that I’m teaching now. 

Grading Scheme

In this course, final letter grades are based on a 1,000 point scale, in which students need only earn 950 points to obtain an A. Obviously with a total of 1,080 points available, it’s quite possible for a student to earn a high grade if he or she simply keeps plugging away at all the various assignments. But this is exactly what I want — for many students, continuous effort will result in improvement across the semester. Constant practice is also makes it more likely that students retain something after the course ends. And students feel better about themselves and their environment with frequent feedback on their performance.

Since this system of assessment makes it more likely that students will be able to demonstrate proficiency by the end of the semester, my grade distribution shifts to the right. Is this grade inflation? I will argue that it isn’t, because the student’s final grade is not based on a hastily thrown together end-of-semester essay that the instructor simply marks as an A or B.