Peer Reviewed Teaching

A couple of years ago, I was part of a pilot project/committee on documenting teaching efforts. The goal was to create a process that mirrored disciplinary research for faculty to use to demonstrate “a peer-reviewed contribution to teaching with real impact in the classroom.” After much deliberation, we called the process a Peer-Reviewed Exploration in Teaching or PRET for short. As the gold standard in research is peer-review, a PRET built on this idea. In this case, the peer group was other faculty, across different disciplines at the university.

As a participant, the process of doing a PRET encouraged quite a bit of reflection. The first part of the process was a proposal that linked specific learning objectives to an activity “that is grounded in pedagogical literature and designed to meet those learning objectives”. Although I’ve always felt that active learning is beneficial, thinking deeply (and putting into a proposal) why this particular activity (I was evaluating my use of the Statecraft simulation) should meet those particular learning objectives was more difficult than I thought. The peer feedback on this step was very useful in solidifying my thinking.

The process included peer observations of my class over a number of sessions – before, during, and after the activity. The observers also conducted a focus group with the students. The report that my peer observers produced from the observations and focus group was eye-opening. It also brought a new level to the debriefing process that we always say is so important with simulations.

Although the PRET formalized this process, I think the basic idea is generalizable and worth the effort. I could see an informal peer-to-peer exchange of proposals and observations work. It encouraged me to think about the goals of the activity in a more systematic way and it provided an outside perspective on whether the activity was meeting these learning goals. As a peer reviewer of other faculty, I learned quite a bit about what active learning looked like in other disciplines and adopted some new ideas.

It’s not what you say…

2934775218_4edd6d67a5There’s a lot to be said for banality. It’s probably the most under-rated of teaching practices, mainly because it’s so little remarked upon. We always talk about pushing our students to the edge of their knowledge and understanding, so that this edge is pushed further back, and we also keep flagging the core ideas as lodestones, but we only rarely come back to the stuff in-between: the logical corollaries of the core concepts.

Yesterday’s class with my negotiation students was a case in point.

The session was centred on the theme of preparation, and asked them to agree a governmental coalition in the wake of the Spanish elections. This threw up lots of great thoughts and discussions about many points: Spanish politics, coalition-building in general, verisimilitude in simulations (they ended up with a grand coalition), and even stuff about preparation (I’d possibly been less than helpful about what prep they needed to do).

But for me, the big lesson was one that I end up discussing at length every year, but never quite manage to embed explicitly in the module’s work. And it’s a banal point: simply put, it’s not what you say, it’s what people hear that matters.

“Sure”, you’re thinking, “that’s pretty obvious, isn’t it?”

Exactly.

Even if you never studied negotiation, then you know enough about constructivism to appreciate the objective weight of subjective interpretations, enough about the importance of clear communication in any sphere of life, and enough about life to know that misunderstandings and talking-at-cross-purposes happens pretty often.

But you also might well have never put those things together to consider the banal point that people will tend to understand things as they understand them, rather than as someone else understands them. So it doesn’t matter if I think I’m being clear, if you don’t think I’m being clear.

Likewise, my students seemed to have a bit of block in understanding why there had been some tension in the negotiations. A couple of groups had left the room to work out some options, and didn’t want to be disturbed by emissaries from the other parties. Unfortunately, since they were the PP and PSOE and ended up with that grand coalition proposal, when they did present it to the others, they didn’t get much joy. Both sides were still quite sore about it, even during the debrief, and we had to work through how this had come to pass before we could get to all that other stuff I mentioned.

I’ve written about this before in a different context and that’s maybe the point: at some stage it becomes so obvious that it’s hard to remember that we need to remember it.

In my case, I’m fortunate that it’s such a pervasive issue that it does always come up at some point in class, but you might not have that. All of us might do well to remember that to leap from central theoretical tenet straight to the boundary can be exciting and engaging, but it can also come with costs.

Horses, carts and my teaching

Bad example
Bad example

It’s really only fitting that the presence of the QAA at work that should drive this week’s reflection and activity. As their audit team spend the week meeting with staff and students, and working through the huge pile of documentation that we’ve given them, we’ve necessarily had to make some adjustments to our usual order of things. In my case, that meant bringing my class to an end a bit early yesterday, so that I could get to my meeting on time.

I’ll admit that when I first got the schedule, I did um and ah about whether I should simply move the class altogether: that would mean less rushing about for me, and more class time for the students. But since I’ve been prepping for this visit for about a year and half, I doubted that the extra time would do anything much to help. This is part of my classic model of exam-stress management: it’s usually too early to stress, until it’s too late to stress. As long as I don’t think about it in the transitional (‘it’s the right time to stress’) period, I’m fine.

Any way, more importantly, I was also thinking about what this particular session with the students was trying to achieve. It’s about negotiation in practice, following on from two sessions on negotiation theory, and is basically there to manage the transition into the much more active phase of the module, by highlighting the difficulties of using theory in practice.

In previous years, I did the following: recapped the key messages from the theory, then showed them a video that I made of a haggle, discussed how I was good and bad in using theory (bad mainly), then sharing my key practical tips on negotiation, then getting them to play a small crisis game (like this one). In short, build linearly from theory through my failings (to make them comfortable with reflective practice), then discussing their practice in the crisis game.

That structure made a degree of sense, but it also is rather gentle in immersing students into what will come, and probably unnecessarily so, on the evidence of all of the cohorts I’ve taught, who seem to grasp the need to look in on their practice and then articulate that externally.

So I chopped it all around. I got them to watch the video before class (also saving me the hassle of IT/speakers/etc.), then launched straight into the crisis game as they arrived in the classroom, debriefed them using the theory, before moving on to discussion of the video, and finally my tips.

This refocusing on their practice meant the session was a much more constructive opportunity for them to consider what they did and do, still with the knowledge that (via the video) I’m also capable of self-critique (not least of my acting/directing skills). Giving them the practical experience at the top of the class really helped to give them something to hang their thoughts on.

The bonus of all this was that it also shortened the session, because the video was moved out, so next year I can work on adding more content/activity to the session.

If there’s a general point behind this, then it’s to think about how and why we scaffold our students’ learning. Often, there’s sound reasoning in building up to more advanced activity, but it’s also worth reflecting on whether that later activity is really any more advanced, or just different from the norm. As I’ve discussed in other contexts, sometimes we need to credit our students with more resilience and capability than we do: isn’t self-reflection something we want to be developing in our students from day one?

You probably don’t have the impetus of a QAA visit to help you try moving things around, but it’s still worth a try.

Can we make our students into superforecasters?

Poster_of_Alexander_Crystal_SeerAs long-standing readers might recall, I read. A lot. But I don’t tend to talk/write about my reading much, either because it’s not immediately relevant or because the rest of my life is so rich with incident. Probably the former, then.

However, I’ve just ploughed through Philip Tetlock & Dan Gardner’s “Superforecasting“, which does offer some interesting perspectives that might be tied to our work in the classroom.

Tetlock will be known to you for the meme that most predictions are no better than chance in their accuracy. As he points out, while that might be true in the aggregate, it’s not true for ever forecaster, and this book (which follows up the Expert Political Judgment that set out the (rigorous) basis for the meme) explores those people who have demonstrated a consistent and measurable ability to outperform both the metaphorical ‘crowd’, but also other mechanisms, such as forecasting markets.

Tetlock’s core point is that such ability is not innate, but learnt and learnable. And that’s what interests me here.

Over the past couple of weeks, I’ve been teaching students about negotiation theory, before I drop them off into the metaphorical deep end of actual negotiation. One of the core skills I’ve been trying to stress to them is the need for preparation, which forms the bedrock of good practice.

However, while it’s easy to say “be prepared”, it’s often hard to know how to prepare, especially if the situation is a complex one. And I can see utility in Tetlock’s approach as a way of improving the chances of having as good an oversight of the preparatory phase as possible.

In practical terms, that means doing things like breaking down intractable problems into tractable sub-problems, looking at the situation from as many different perspectives as possible and engaging in constant checking of your practice and biases.All the kinds of things we might be encouraging students to do as active learners, in fact.

And perhaps that’s the point. Learning is a memetic process, where we draw analogies across from one place to another, hopefully casting light on the way. Tetlock’s not interested in this book about effective actors, although he does mention several examples of how his core ideas get operationalised, but about our ability to forecast events. I’m less interested in that, but can see how I could use it to improve students’ performance elsewhere.

Tetlock talks about the rise of the ‘mission command‘ model in the military, the notion that what matters is the underlying intent, which local commanders can then find the most appropriate way to achieve it. This replacement of the strict hierarchy of command-and-control with something much more flexible and adaptable is not so different from the shift towards active learning. We set out learning objectives and students find their own path towards them.

I wouldn’t want to stretch this too far, but it makes the point that we should be casting our nets as widely as possible when developing our practice, because the analogies can take us to places that might never have considered before.

Cleaning up other people’s mess (in a good way)

3175714614_321ac67ede_bA few weeks ago, I wrote about simulating the Greek crisis. I suggested then that one issue in doing this was the difficulty of carrying things over from year to year in the classroom: students change, curricula change, you never quite know whether it’s still going to be relevant, etc.

As Amanda rightly pointed out in an email to me some time later, you can perfectly well do it, with a bit of thought. So it’s with that bit of thought that I am now doing it.

Last year I created my first online asynchronous simulation for the INOTLES project in which I participate. As you’ll see from the post, it’s a simplified recreation of the East European situation, with a friendly (if ponderous) EU-like structure on one side and a confident (if worryingly so) Russia-like country on the other.

I played this with my students too, with the upshot that the ‘Russians’ produced a surprising success in sealing a deal with the ‘East Europeans’ (largely over a misunderstanding, but let’s not pretend that doesn’t happen in real life too). I put the simulation back on the shelf, mused on what had happened and then basically forgot about it.

Until Amanda’s email. There’s no reason why this year’s students can’t pick up where their predecessors left off.

It’s a fictional scenario, with all the requisite information provided. Since it allows for a wide-ranging set of actions, there is no obvious end-point or stable equilibrium. Indeed, one might imagine that some students might take the opportunity to revise the actions of the past, just because they can. Certainly, given the rather devil-may-care approach to a second round of the Hobbes games in class yesterday, that looks like a rather likely outcome.

I’ll quote Amanda at some length here:

Whenever we do a simulation, it tends to be a new run of an old game–how neat would it be to have the simulation just continue, with students acting as the newly appointed representative for that country and having to work with old agreements produced by students who are no longer in power?  I find the idea really interesting, not only for the sense of realism it brings to ongoing negotiations, but also for the real-world skill of having to step into a job vacated by someone else and having to figure out what the prior office holder did and how to incorporate their decisions into your own

Amanda’s last point is perhaps the crucial one: we all have to pick up other people’s stuff and deal with it – it’s a basic stable of professional life – so getting to experience that is a useful opportunity for personal development.

Indeed, in this game the original conceit that it opened with no particular situation is clearly unrealistic, so we’ll learn about path dependency directly.

Amanda’s one concern was about record-keeping: how to capture what had happened, so that we can pick it up again. Well, I’ll admit that this isn’t a big issue in this case. The final agreement reached ran to a full four lines of hand-written text and there was nothing else to share. I’m hoping that this time around we’ll have clearer sight of the next year, so that paper-trails can be left, with all the joys that brings.

As usual, this is all new territory for me, so I’ll be reporting back as we progress.

The trouble with measurement

IMAG4008So it turns out I’m a Thinker. And a Doer. And a Planner. And a Solver.

Last night, I was administered the learning styles questionnaire of my eldest, which had come home from school earlier on, which revealed that I was a very balanced individual. Possibly.

Now it could be that I am very balanced. Or that my enormous experience and modesty mean that I have learnt to appreciate and internalise these different approaches. Or that the questionnaire is not very good at distinguishing between them. Or that learning styles is possibly unhelpful as a model for understanding individuals.

Of maybe it’s just because I didn’t tick number 33: “I’m fun to be around.” Who knows?

And that’s actually the issue we confront very often in L&T: we just don’t know what we’re measuring and whether it’s meaningful. It’s easy to pock holes in a questionnaire for kids, but it’s just as common an issue in universities, where all these clever people sit around working hard to describe clever ways to resolve precisely such issues.

I’ve observed before that students’ evaluations of modules/courses – which we set such great stock by in the UK – can only ever be part of judged the quality and effectiveness of our teaching. “Have we made a good impression on our students?” is a different question from “have they learnt something useful?”, yet we often conflate the two.

Likewise, as has been noted on this blog many times, we talk about new pedagogies and why they’re great, without much evidence to back it up. Just because I think simulations are great, doesn’t mean they’re great, and my anecdote doesn’t match systematic research of outcomes.

So what to do?

Partly, we have to rein in our enthusiasm and be frank about the limits to our knowledge. I think I speak for all the ALPSblog authors when I say that we talk as much about the costs and limitations of our active learning practice as we do about the benefits: beware anyone who offers you the moon on a stick.

Partly, we have to go out and get that evidence. For my part, I’m going to use some of the money the nice people at the Higher Education Academy gave me to conduct some research into simulations to do just that. I’m still working on the details, but it would seem to be a worthwhile activity, on a number of levels.

To return to a theme we discussed not so long ago, if we are to bring more people along with us in improving pedagogy, then we have to make the case better.

You – as a Thinker, Planner, Doer and Solver – can be part of that too.

It’s just a game…

michael-winner
If this means nothing to you, then rejoice.

We talk a lot about the immersive aspect of simulations on this blog: the way in which students become participants, actively taking part in the learning environment and so internalising the material.

However, it’s also important to recognise that this is not without issues, something that I was reminded of with this week’s news about the arrests made in the run-up to the Pokemon world championships. While that case wasn’t directly about immersion, it did speak to me of a problem of perspective and proportion. As much as we amplify the learning, so we also amplify the potential for distress.

I wrote an article some years ago about enhancing immersion, which discusses this in much more depth, but it’s helpful to reprise some of the key messages that flow from that about managing it.

  1. Neither staff nor students should lose sight of the simple fact that the simulation is just that, a simulation. As much as immersion is to be encouraged, so too must be an awareness of the artifice. Beforehand, you might warn students about the potential for getting drawn in. During the game, students can be required to perform tasks that make them step out of their roles (keeping notes and reflections on their actions, for instance). Most importantly, afterwards there must be a debrief, where students are consciously and conspicuously brought back to the ‘real world’. This is also a good point for some discussion about the artificiality of the simulation.
  2. There is a delicate balance to be struck between having too much conflict in a simulation and too little. Conflict helps to generate buy-in, as students have to defend their position and attack others’; it gives a easy way into them feeling that something is at stake. However, if that conflict becomes too big a part, then it colours everything else that happens, including life outside of the game. Similarly, if the impression is that everyone agrees, then students can struggle to get motivated by the topic. Usually, it’s pretty obvious if there’s enough conflict to work, but if you’re not sure, then I find that it’s good to sit with a colleague and talk through the issues with them;
  3. Simulations are artificial, but they still are supposed to tell us something about the real world, so it’s important not to completely box in the experience. This might seem to go against what I’ve just said, but it’s actually an extension of it: discussing the artificiality also requires us to discuss the realism. Certainly, I can say that I have learnt a lot about a lot of people, just through interacting with them on a simulation: not always good stuff, but definitely stuff that helps me to understand them better. The key point here is – once again – context.

In summary, simulations offer us great potential for insight – into ourselves and others – but we have to handle that with care and thought.

Modelling a mess

2453997644_8673f7da14_o
Some Greek misunderstanding about having a meal break

One of our perennial themes here at ALPSBlog is the notion of learning from learning, the idea that even when nominal objectives are not achieved, there’s still something (much, even) to be learnt. I get you to try and build a 20m bridge across a lake, using only tissue paper, you fail, but get something from the experience none-the-less.

This past week I was asked (jokingly, I hope) whether I could capture the negotiations on the Greek bailout saga. I’ll assume you’ve heard about this, even if you didn’t follow the details and that, like me, your over-riding impression was one of a chronic failure to find constructive or timely agreement between the various parties. The Onion has its own drole simulation of the forces at work.

Now this is different from the learning from failure model in that it appears to be a situation where failure to find some agreement is not an acceptable option and where all the good options are excluded and all the remaining options come with big costs. Let’s call it a messed-up situation that only gets worse over time.

Now this is really interesting to explore with students, both as a specific case and an example of poor negotiation. As a case-study, the Greek crisis touches on democracy, European integration, party politics, macro-economics and a bunch of other big social science issues: any simulation would be a great entree into these. As a negotiation format, it brings together an interesting combination of domestic and international actors, with varying degrees of resource (and legitimacy) in an environment where rules are highly flexible (which means you could capture much of the fluidity in a classroom).

However, the big question is quite what you want to capture in a simulation.

If the main thing is the inability to find a lasting solution, then you immediately hit an issue with the time and space constraints of the classroom: neither your students nor your university will let you have a simulation that drags on for several years. The only option here would be to have a game where resolution was fundamentally impossible, and each new cohort of students would simply play during their allotted time, to reach a new conditional outcome. However, that comes at a price to the visibility of the whole sequence.

If the aim is to highlight the lack of good options, then it becomes much easier. A few years ago (i.e. nearer the start of the Greek crisis), I wrote a game about austerity politics, modelled very much on the Eurozone’s travails. This game hasn’t got a good outcome (in the sense of optimal for most players), but it could be extended further, either by adding more domestic political constraints/challenges or by creating an external IMF-ish role to make it even more painful. The game already has rounds, so you could just keep on pushing this on, so that everyone get’s even more tetchy with each other.

A third option would be to focus on how the unstructured nature of the negotiation environment matters. Students could be given the task of trying to design a more formalised structure for the participants, with roles and responsibilities, in order to help them see how the different parts fit together. This would then be a critique of the real situation and a way into the underlying tensions.

This last is something that I’m tempted to try with my students this autumn, not least because they will have to learn how to organise themselves in order to be able to discuss productively how others should be organised. Since I’m aware that my students do read these posts, I’ll say no more, other than I can see several ways that they could do this, to better or worse effect.

This last point brings us back to the main issue, namely of identifying your learning objectives. That’s not just true of this, but of all your learning activities. Once you know where you want to end up, the path to get there becomes that much more clear.

 

Guest Post: European Studies for Business Students (Natalia Timus)

 

timusToday’s guest post comes from Natalia Timus (Maastricht):

One of the challenges of teaching European studies apart from its multidisciplinary character, is a wide target audience interested in this field of studies. This requires a certain degree of flexibility on the part of the instructor in adapting their knowledge and skills to be transferred during the learning process to the specific audience, be it political scientists, historians, economists, or lawyers, for example.

For the second year in a row I have been invited to teach a crash course on the ‘EU and its outsiders’, with a focus on Eastern European neighbourhood, for a summer program of a business school. This represented a new adventure for me and a major challenge for my teaching approach. I was well aware that I have to keep it simple, interactive, and get most of the things done during the classroom, since the course consisted of three days intensive teaching (15 hours) and everyone was looking forward for some beach or sightseeing time in Nice, France.

My first concern in defining my teaching approach was how to optimally talk about business and economic developments from the perspective of a political scientist that I am, and not sliding into the business field that might represent a big mystery to me. For the beginning, I considered important identifying major issues within the field of political economy that might provide a fertile ground for the study of business and political science. Taking into account the multinational profile of my students, as well as the high probability of their lack of any previous background on Eastern Europe or EU external relations, I decided to balance between the time devoted to cover the domestic level transition to democracy and market economy and the external influence, covering both the EU  and Russia.

Now that half of the job has been done, I still had to define the appropriate pedagogical approach I would use. Although I knew little about students’ background, I was informed that there is a large majority coming from the United States higher indication system, but half of them were Chinese, many of them being enrolled both at the BA and MA levels within business calls, some of them were practitioners that decided to use the summer as an opportunity for personal development, and their age varied from early twenties to forties. There was also the division between native and non-native English speakers that I had to be taken into account. In these circumstances, I reflected upon the pedagogical strategy that would provide enough room for everyone to feel comfortable raising questions and taking part in class activities, as well as balancing between individual and team work.

Knowing that case study is a widespread pedagogical approach within the business education, I have conducted a small research for better understanding the use of this method and ended up adapting it to my specific content to be delivered and the time constraints. I chose to work on four case studies engaging students in face-to-face group work during the classrooms and devoting around one hour for each case study.

Apart from that, being a fan of debates, which I have acquired as a learning approach during my high school years, I also inserted two open class debates, where individual student participation was assessed, and one final debate, like a desert for the course, which has been structured as a parliamentary debate and the two groups shared the grade based on their performance.

As I was new with teaching EU studies to business students, their feedback was very important to me. Therefore I have devoted 5 to 10 minutes for debriefing after each day for a joint reflection on the teaching of the learning process, and the final debriefing when I asked the students to reflect on two questions: 1) what course element did you like the most and why? and 2) what could be further improved and how?

I was glad to see that the students felt comfortable enough with providing an open face-to-face feedback additional to the formal evaluation forms that they had to provide to the administration. Overall, they shared the view that the course was well paced, with a good balance between lecturing and interactive parts, as well as regular breaks of five minutes once an hour. They also appreciated the case studies and their class material that they had to prepare on the spot and then take part in group and class discussions. Some of them suggested as a potential improvement a more focused lecture on Russia, taking into account their personal interests in the country, or would have welcomed more in-depth information on the EU, which largely  represented a mystery for them. But all of them liked the fact that there was enough room to ask questions and that I, as an instructor, was feeling comfortable with addressing them, even if they were going somehow beyond the focus of the course.

I must confess I truly enjoyed this unique experience although it required a considerable amount of time and effort for preparing the tailor-made course for teaching the ES to business students. I felt empowered by their positive feedback and it motivated me to improve further for the second year and get more comfortable with my teaching approach. And finally, it just feels special when one as an instructor has the opportunity of introducing for the first time some knowledge to the students and know that it will have a significant impact on their personal development and the way they perceive the world in the future.

Book Review: The New Science of Learning

New Science of LearningWe’ve talked previously about Classroom Assessment Techniques by Thomas Angelo and K. Patricia Cross (here, here, and here), so I thought I’d post a review of another book:

Terry Doyle and Todd Zakrajsek, The New Science of Learning: How to Learn in Harmony with Your Brain, Stylus Publishing, 2013.

The New Science of Learning is a very concise and easy-to-read advice guide for undergraduates that is based on the findings of cognitive science research. I’ll be using it this fall in my first-year seminar. My hope is that it will help students, many of whom are not that well prepared for college, improve their academic performance. Here is one of the book’s authors speaking at Quinnipiac University, from the perspective of how to teach more effectively.

I’ve created these writing assignments that correspond to the chapters of the book:

  • Of the different practices that help people learn more effectively, which is the one that you currently use the least frequently? What would you need to change in your life so that you used it more frequently?
  • Think about the last three nights. How well did you sleep on each of these nights? What changes would enable you to sleep better? How can you implement these changes?
  • Thinks about the last three days. At what times were you physically active and for how long? How did your levels of mental alertness change during the day? Do you notice any pattern between physical activity and alertness?
  • Do you write notes by hand in your college courses? Do you annotate text that is assigned in these courses? Why? Given the benefits on learning of note-taking and annotating reading assignments, how well will you perform academically this semester? 
  • Describe an assignment in one of your courses this semester that reflects the pattern recognition principle of similarity/difference, proximity, figure-ground, or cause/effect. What is the assignment and how does it reflect the principle? What will be the effect on your understanding or memory of the material?
  • Name an activity in which you use either distributed practice or elaboration. What is a specific change that you can make in your daily behavior to better incorporate either one into your college experience?
  • What is your approach to failure? Do you embrace the possibility of it or try to avoid it at all costs? When you fail, what is your reaction? Based on your answers to these questions, do you have a growth mindset or a fixed mindset toward learning? Why?
  • Do you engage in task shifting? When? What is a specific change that you can make in your daily behavior to reduce task shifting?