Can we make our students into superforecasters?

Poster_of_Alexander_Crystal_SeerAs long-standing readers might recall, I read. A lot. But I don’t tend to talk/write about my reading much, either because it’s not immediately relevant or because the rest of my life is so rich with incident. Probably the former, then.

However, I’ve just ploughed through Philip Tetlock & Dan Gardner’s “Superforecasting“, which does offer some interesting perspectives that might be tied to our work in the classroom.

Tetlock will be known to you for the meme that most predictions are no better than chance in their accuracy. As he points out, while that might be true in the aggregate, it’s not true for ever forecaster, and this book (which follows up the Expert Political Judgment that set out the (rigorous) basis for the meme) explores those people who have demonstrated a consistent and measurable ability to outperform both the metaphorical ‘crowd’, but also other mechanisms, such as forecasting markets.

Tetlock’s core point is that such ability is not innate, but learnt and learnable. And that’s what interests me here.

Over the past couple of weeks, I’ve been teaching students about negotiation theory, before I drop them off into the metaphorical deep end of actual negotiation. One of the core skills I’ve been trying to stress to them is the need for preparation, which forms the bedrock of good practice.

However, while it’s easy to say “be prepared”, it’s often hard to know how to prepare, especially if the situation is a complex one. And I can see utility in Tetlock’s approach as a way of improving the chances of having as good an oversight of the preparatory phase as possible.

In practical terms, that means doing things like breaking down intractable problems into tractable sub-problems, looking at the situation from as many different perspectives as possible and engaging in constant checking of your practice and biases.All the kinds of things we might be encouraging students to do as active learners, in fact.

And perhaps that’s the point. Learning is a memetic process, where we draw analogies across from one place to another, hopefully casting light on the way. Tetlock’s not interested in this book about effective actors, although he does mention several examples of how his core ideas get operationalised, but about our ability to forecast events. I’m less interested in that, but can see how I could use it to improve students’ performance elsewhere.

Tetlock talks about the rise of the ‘mission command‘ model in the military, the notion that what matters is the underlying intent, which local commanders can then find the most appropriate way to achieve it. This replacement of the strict hierarchy of command-and-control with something much more flexible and adaptable is not so different from the shift towards active learning. We set out learning objectives and students find their own path towards them.

I wouldn’t want to stretch this too far, but it makes the point that we should be casting our nets as widely as possible when developing our practice, because the analogies can take us to places that might never have considered before.

A Quick Exercise on Confirmation Bias and Hypothesis Testing

This neat exercise featured on the New York Times takes a few seconds to play and includes a neat set of examples of how confirmation bias impacts government policy and corporate America.

Basically, you are presented with 3 numbers in a sequence, and asked to guess the rule that governs the sequence. You can enter in any 3 numbers you like, and the system will tell you whether or not your sequence follows the rule or not. When you are ready to guess, you enter it in but you receive no second chances. Apparently 78% of people make a guess without getting a single ‘no’–and most get the rule wrong.

The example in the NY Times is ‘2, 4, 8’. A number of possible rules could come to mind–must contain multiples of 2, or even numbers, or that the number doubles the one before it. The actually rule in this case is even simpler: the number must be larger than the one before it, meaning that ‘4, 8, 16’ works, but so does ‘1, 10, 3593’.

They don’t mention it in the article, but this exercise can adapted to teach hypothesis testing. Used in class, you can put the sequence on the board and have students suggest other sequences, which you then judge as either following or not following the rule. They have to use this information to come up with the right answer.

If this sounds familiar, it may be because one of the very first entries on this blog was about the board game Zendo, which does precisely this, only with physical pieces rather than numbers. I still use Zendo on day 1 of my methods class, and find it a really useful tool for teaching a variety of methodological skills. This numerical version is a great, easy activity to pull out for a quick fix on helping students with their logical thinking.

Fear and Loathing of Broccoli

Fear and LoathingI very rarely incorporate feature films into my courses — Dr. Strangelove has been one of the few exceptions — but anyone who is teaching political psychology should take a look at Inside Out from Pixar. The film, for which the psychologists Paul Ekman and Dacher Keltner functioned as consultants, is a visual representation an 11-year old girl’s mind. Emotions take the center stage, especially in regards to memory formation and retrieval, but imagination, attention, and other processes also play into the film’s plot. More detail on the science incorporated into Inside Out is here.

Confirmation Games

KatnissThe New York Times recently published an interactive illustration of confirmation bias: guess the rule obeyed by a sequence of three numbers. I won’t go into detail about the game other than to say that it’s wonderfully simple and I definitely fell into the mind trap. Some implications for politics and business are presented after players submit their answers and this can provide a launch point for class discussion.

The puzzle nicely complements the Zendo game in which players create hypotheses about an arrangement of blocks — by demonstrating the cognitive biases that affect much of our decision making.

The Best Laid Plans

Cat and MouseA while back I wrote a series of posts on reworking my first-year seminar. My assumption was that this fall’s version would meet three days a week, as happened in the course’s initial iteration. I recently learned that instead it will meet on Tuesdays and Thursdays. Given that much of the course involves student-to-student interaction in the classroom, the new schedule necessitated further changes. To start, I dropped the book that I had originally fit into the last third of the semester, and with it plans for a class-wide Twine project. The course now looks like this:

  • Team-based Twines on the book An Ordinary Man (Rwanda).
  • Simulation exercises on the first four cases in Chasing Chaos (Rwanda, Sudan, Sri Lanka, Sierra Leone).
  • Team-based Twines on the last case in Chasing Chaos (Haiti).

Since this is a course for incoming college students, I added The New Science of Learning and some other meta-cognitive content on skills for academic success. This means that students will have on average three writing assignments on readings per week even though the class only meets twice a week, which I think this is a good thing. Students won’t be able to forget about the course between Thursdays and Tuesdays.

As I discussed in my informal assessment back in January, I had a problematic formulation for the briefing memo that prepared students for each Chasing Chaos simulation. I’ve rewritten the assignment instructions accordingly, and created a new sample memo for students to use as a guide. The effort that I’m putting into the design of this course reflects something about how college works that I’ll discuss in more detail in my next two posts.

Book Review: The New Science of Learning

New Science of LearningWe’ve talked previously about Classroom Assessment Techniques by Thomas Angelo and K. Patricia Cross (here, here, and here), so I thought I’d post a review of another book:

Terry Doyle and Todd Zakrajsek, The New Science of Learning: How to Learn in Harmony with Your Brain, Stylus Publishing, 2013.

The New Science of Learning is a very concise and easy-to-read advice guide for undergraduates that is based on the findings of cognitive science research. I’ll be using it this fall in my first-year seminar. My hope is that it will help students, many of whom are not that well prepared for college, improve their academic performance. Here is one of the book’s authors speaking at Quinnipiac University, from the perspective of how to teach more effectively.

I’ve created these writing assignments that correspond to the chapters of the book:

  • Of the different practices that help people learn more effectively, which is the one that you currently use the least frequently? What would you need to change in your life so that you used it more frequently?
  • Think about the last three nights. How well did you sleep on each of these nights? What changes would enable you to sleep better? How can you implement these changes?
  • Thinks about the last three days. At what times were you physically active and for how long? How did your levels of mental alertness change during the day? Do you notice any pattern between physical activity and alertness?
  • Do you write notes by hand in your college courses? Do you annotate text that is assigned in these courses? Why? Given the benefits on learning of note-taking and annotating reading assignments, how well will you perform academically this semester? 
  • Describe an assignment in one of your courses this semester that reflects the pattern recognition principle of similarity/difference, proximity, figure-ground, or cause/effect. What is the assignment and how does it reflect the principle? What will be the effect on your understanding or memory of the material?
  • Name an activity in which you use either distributed practice or elaboration. What is a specific change that you can make in your daily behavior to better incorporate either one into your college experience?
  • What is your approach to failure? Do you embrace the possibility of it or try to avoid it at all costs? When you fail, what is your reaction? Based on your answers to these questions, do you have a growth mindset or a fixed mindset toward learning? Why?
  • Do you engage in task shifting? When? What is a specific change that you can make in your daily behavior to reduce task shifting?