Process v. outcome in simulations

Chad's simulations were becoming a bit too realistic...
Chad’s simulations were becoming a bit too realistic…

Very occasionally, different parts of one’s life collide with each other, often after you’ve had a bit too much to drink, which further contributes to the further unrolling of the evening.

Any way.

In this case, no drinking (except of cups of tea) was involved for me this week, when I attended a ‘wargame’ of the British renegotiation of European Union membership. Run by Open Europe, the day had two parts, each intended to cast some light on what might happen in, respectively, the current renegotiation and then in the event of a ‘no’ vote in the referendum.

This was a high-rent production. Fancy City venue, live-streaming of the entire event, plus (most importantly) roles being played by People (former ministers and ambassadors): The British were represented by a former Foreign Secretary, Maclom Rifkind, and a former Chancellor, Norman Lamont. All of this built on a previous event in 2013 and given the number of TV camera crews from across Europe, it was not your usual event.

For all that, it was also a very strange affair.

Continue reading “Process v. outcome in simulations”

How innovative are we?

Me, back in my punk phase.
Me, back in my punk phase.

Around these parts, we tend to make a lot of innovation. We write posts about cool new things we do, partly because we like telling people about such stuff, partly because we think you’d like to such stuff too.

Of course, we’re also very sensitive and certainly don’t fetishise innovation: it’s got limits and there’s a time and a place for it all. We’re not going to make you do something you don’t want to, or need to.

But there’s another question that sits to one side in all of this: just how innovative are we in any case?

Continue reading “How innovative are we?”

Horses, carts and my teaching

Bad example
Bad example

It’s really only fitting that the presence of the QAA at work that should drive this week’s reflection and activity. As their audit team spend the week meeting with staff and students, and working through the huge pile of documentation that we’ve given them, we’ve necessarily had to make some adjustments to our usual order of things. In my case, that meant bringing my class to an end a bit early yesterday, so that I could get to my meeting on time.

I’ll admit that when I first got the schedule, I did um and ah about whether I should simply move the class altogether: that would mean less rushing about for me, and more class time for the students. But since I’ve been prepping for this visit for about a year and half, I doubted that the extra time would do anything much to help. This is part of my classic model of exam-stress management: it’s usually too early to stress, until it’s too late to stress. As long as I don’t think about it in the transitional (‘it’s the right time to stress’) period, I’m fine.

Any way, more importantly, I was also thinking about what this particular session with the students was trying to achieve. It’s about negotiation in practice, following on from two sessions on negotiation theory, and is basically there to manage the transition into the much more active phase of the module, by highlighting the difficulties of using theory in practice.

In previous years, I did the following: recapped the key messages from the theory, then showed them a video that I made of a haggle, discussed how I was good and bad in using theory (bad mainly), then sharing my key practical tips on negotiation, then getting them to play a small crisis game (like this one). In short, build linearly from theory through my failings (to make them comfortable with reflective practice), then discussing their practice in the crisis game.

That structure made a degree of sense, but it also is rather gentle in immersing students into what will come, and probably unnecessarily so, on the evidence of all of the cohorts I’ve taught, who seem to grasp the need to look in on their practice and then articulate that externally.

So I chopped it all around. I got them to watch the video before class (also saving me the hassle of IT/speakers/etc.), then launched straight into the crisis game as they arrived in the classroom, debriefed them using the theory, before moving on to discussion of the video, and finally my tips.

This refocusing on their practice meant the session was a much more constructive opportunity for them to consider what they did and do, still with the knowledge that (via the video) I’m also capable of self-critique (not least of my acting/directing skills). Giving them the practical experience at the top of the class really helped to give them something to hang their thoughts on.

The bonus of all this was that it also shortened the session, because the video was moved out, so next year I can work on adding more content/activity to the session.

If there’s a general point behind this, then it’s to think about how and why we scaffold our students’ learning. Often, there’s sound reasoning in building up to more advanced activity, but it’s also worth reflecting on whether that later activity is really any more advanced, or just different from the norm. As I’ve discussed in other contexts, sometimes we need to credit our students with more resilience and capability than we do: isn’t self-reflection something we want to be developing in our students from day one?

You probably don’t have the impetus of a QAA visit to help you try moving things around, but it’s still worth a try.

What makes for ‘quality teaching’?


British universities are now all back up and running for the academic year, after our traditional summer break, which we like to use laughing at American colleagues who had to start back in August (August!!).

As pay-back for such rude behaviour, we’re also now about to receive the government’s latest thinking about Learning & Teaching, in the form of a Green Paper.

No one is sure what it’ll contain, but it’s likely to set out plans for a Teaching Excellence Framework (TEF), linked to student funding and to assorted other reforms of the sector. I’ve written about TEF before, but I’d like to return to the theme, since it looks set to become the defining feature of L&T in this country for the next decade or more, just like its research counterpart.

Part of the thinking that now seems to be floating around is a progressive building of TEF from a starting point of basic metrics (progression, completion, good degrees, employment, etc.) into something with more of a qualitative element. It’s this second part that I’d like to consider some more.

Personally, I find the concept of ‘high quality teaching’ rather problematic. How does one judge such a thing? Is it about the quality of the outputs, or the inputs, or the improvement between the two? Does it need to be grounded in pedagogic research, or is it sufficient that ‘it works’? Do we judge the individual elements of learning & teaching (whatever that might mean) or the whole package (whatever that means too)? Do we stick to the formal curriculum or to the softer parts of a university experience?

All are valid questions to ask, and all provide very different answers and incentives to act in a particular way.

Consider a challenge that my home department faces. We tend not to get a terribly high percentage of first class and upper second degrees (‘good degrees’ in the jargon), compared to the rest of the university. It’s not a big difference, but enough to be noticed. One of the main reasons that we’ve identified is that we use a wide range of forms of assessment compared to others: not just essays and exams, but also presentations, group projects, negotiating briefs, policy briefs, reflective pieces, and so on. We do that because we believe it develops more useful skills for our students, since they are much more likely to encounter those kinds of things in the outside world.

The problem is that our range of assessment means that students don’t get the depth of practice that they would if they only did essays and exams, which in turn tends to depress their grades. Having seen examples of such assessment monocultures (bicultures, more precisely), I can attest to their value in maximising performance in those assessment forms.

So the question is whether that’s a good thing or a bad thing. And one could see how the argument could be made both ways. The real challenge comes in working out how to balance different claims in any evaluation exercise.

The riposte to this would be that the combination of quantitative and qualitative elements in TEF would exactly allow for universities to contextualise and clarify their situations. But that’s not the same as offering a mechanism for balancing out the different parts, not least for the inevitable league table that will result from all of this.

As I discussed last time, I feel rather conflicted about TEF. On the one hand, it’s an excellent opportunity for L&T to be brought to the foreground and given the attention it requires and deserves. On the other, I worry that it will tend to push us back into small-c conservatism of practice, where innovation and risk are discouraged, because of the potential upsetting of the basic metrics that will weigh heavily in the exercise. We’ve seen something similar with REF, where the gaming of the rules is the dominant mode.

If there were a way to embed an incentive to challenge ‘traditional’ practice and to create spaces for trying out new ideas, then I’d be very happy, concerns over innovation for innovation’s sake notwithstanding. Even just something that required us to justify the choices we make would be a start, maybe in the form of a statement by each teaching unit of its pedagogical underpinnings. Much of it might be rather vapid stuff to start with, but the exercise would at least force people to engage with the research and literature that’s out there and get that ball rolling.

Another possibility might to require universities to host a minimum number of L&T testbeds, to try out new approaches, with their outcomes ring-fenced from the metrics, so as to allow for some risk-taking and to generate internal potential for more wide-spread change. Certainly, there’s an issue of how students might feel being used as guinea-pigs, but that’s manageable.

These are just random thoughts, but I hope they point towards some of the tensions that have to be resolved with TEF. That the Green Paper has been delayed suggests that the politicians have not agreed on what should be included: academics need to be ready to engage and contribute to shaping whatever comes out.

Who’s afraid of active learning?

Low-tech, high impact (and colourful)
Low-tech, high impact (and colourful)

When I talk with colleagues who teach, but who aren’t really into teaching (if you know what I mean), there’s often a sense that the L&T crowd are out to make life difficult for them. Regardless of whether that’s true or not (as discussed last week), the image is one of ‘us’ wanting to make ‘them’ do ever more complicated things, for no good reason.

This is actually a big issue, but for now we’ll focus on some parts of it and see how we go. Within that, one issue is what I will be calling the promotion/representativeness problem*. This is just the tension between us trying to share our work and often picking out the most glamorous or extensive example, rather than the simple thing that might be of more general interest.

To give you an example, for all my work on simulations, I’ve have just as much positive feedback (and adoption, a more useful measure) of my ABC post-it note technique.

The problem is that the things we talk about – the things that we have often put a lot of time and effort into, to be sure – are not the things that colleagues might find most useful. To use a more pertinent example, I’m giving a workshop on active learning later this week and I’d like to recommend a great demonstration of this, but it’s being run by the university down the road and it certainly wouldn’t work in two places so close.

All of this is a more general issue in talking about active learning. It doesn’t have to be huge simulations, or totally re-creating a curriculum; it can be a pile of small activities, bundled in with what already happens.

Of course, the solution to this is to go all jujitsu and use weaknesses as strengths.

If we’re in the game of trying to colleagues that L&T is as important as research – which I’ll suggest is our usual tussle – then why not make the L&T look more like research?

Students-as-Researchers is not only a great handle, it can also be great work. And it doesn’t have to be about working to formal outputs, although you can do that: it can be about getting students to engage with the research process in more explicit ways. And that can start from the first semester: last year, I got my Liberal Arts & Sciences students to present a research proposal for their first piece of assessment and used that as a way to help them get into the mindset of research. The projects might not have got funding in the real world, but as a first step, it was very helpful indeed.

Maybe the way to approach all of this is to treat it as we would a class: first question is – as always – what do we want to achieve?

If we talk to colleagues and get a proper sense of what they look for students to learn, then we can offer something from our range of pedagogies and techniques that might suit them, rather than throwing out idea after idea at them in the vague hope that they spot something they like.

That’s not about jargon, and it’s not about the fanciest thing you’ve ever done, and it’s a slog, but in terms of securing lasting improvement in our collective practice, it’s probably the best way forward.


    • I’m thinking that if I dress it up a bit, then it’ll sound like something that the research-focused might consider**

** – obviously, such dressing-up is also a problem, for general accessibility

Proxy, proxy, proxy: out! out! out!

Right place, right time, wrong conversation

At some point we must have run out of conversational topics on holiday, because I found myself on a beautiful Croatian beach, trying to explain the ‘good degrees’ metric to my wife. To my credit, I didn’t try very hard, although it wasn’t helped by the fact that I struggled to come up with a good justification for it.

For non-UK readers, “good degrees” measures the percentages of awards in the top two categories (firsts and upper seconds) for undergraduates. That’s about 70% of them, a figure that’s risen over time.

My Croatian struggle came in trying to unpack what it showed and why that might be useful.

Partly, it’s a reflection of our student body: if we have more academically-able students, then they’ll be more likely to get better grades, and so a ‘good degree.’

But equally, it’s also a reflection of our assessment and grading. Are we the kind of place that rewards good work (assuming that student populations are broadly equivalent)?

So what does it show? Our selectiveness? Our accommodating assessment? Our great students? Something else?

The issue is one that might have remained of minimal interest to anyone, but for the use of the metric in league tables. These aim to rank universities on the quality of their provision and have become an important part of the British HE sector, not least for recruiting new students. However, that also introduces an incentive to improve one’s ‘good degree’ metric.

The obvious responses in this situation would be to have stricter entry requirements (which is also a league table metric) and to have more lenient assessment. The first is very much in the universities’ hands and you’ve seen exactly that happening. The second is a very different kettle of fish.

Assessment in the UK differs from many other countries with its use of second- and external-examining (see this for more), which is in turn bound by a national framework of performance standards: broadly speaking, any one at a UK university should be marked to the same kind of standard, whatever and wherever they study, against benchmarks, rather than against each other. And, broadly speaking, that’s what happens.

The upshot is that any (league table) pressure to go soft on assessment is counter-weighted by the need to meet sector and disciplinary norms: as our university tells us, the norms are the norms and we shouldn’t be messing around with them.

The problem now becomes one of still wanting to improve ‘good degrees’, but losing the most obvious tool to achieve that. So instead, we try other things. These cover pretty much every part of what we do in our teaching. Are there pinch points in assessments or modules that we can redesign, to meet learning objectives in a more accessible way? Are there staff how need help with redesigning assessments or with providing support? Are there early markers of weaker performance that can trigger timely intervention? Are there extra resources we could provide that would help improve student outcomes?

All of those questions are good questions to ask of any educational body, since they all speak to the core question of ‘how do we help our students to achieve their full potential as learners?’

Unfortunately, all of those questions are not ones that convert into nice, simple metrics. Instead we fall back onto things like ‘good degrees’ and back into the question of what we are trying to do.

The issue is an old one: if you make something a metric, then people tend to work towards those metrics, they gain value in of themselves. That’s a problem if we forget that metrics are proxies, which is all too easy to do.

As we move towards a Teaching Excellence Framework here in the UK, that’s something we’re going to have to deal with more and more.

Eppur si muove

Politics UOS-6056aIt’s the end of the academic year for us, which mainly means wrap-up meetings, graduation ceremonies and preparation for the autumn. It’s also time for the “we’re not like school teachers, so we don’t get two months’ holiday” conversations with people.

As such, it’s also a good time to have a bit of a think about how things have gone and how they might change in future. As you’ll know from my last post, I’ve got plenty to think about this summer, in general, but I’d also like to talk a bit about my department in particular.

Earlier this year, the university undertook a major review of its operations, which included proposals to reduce the number of staff in Politics. While the final reduction was not as large as originally proposed, we do still have to provide effectively the same portfolio of activity in learning & teaching as this year, with fewer people.

I leave to others to discuss the wider context, but it is perhaps useful to talk a bit about how we have dealt with this situation, since it is (sadly) not uncommon.

A number of points present themselves as worth sharing here.

The first is that it is always worth considering whether everything you offer needs to be offered. As a social science, optional modules/courses/units tend to be plentiful, as colleague X arrives and says they could do something fascinating on Y, it runs for some years until X leaves and you somehow feel that Y is central to your provision.

Having sat on the QAA Benchmark group last year, I feel reasonably confident in saying there’s no consensus at all about what a ‘Politics’ degree must contain. Some methods, theory and core concepts certainly, but otherwise it’s open.

As long as you can defend the quality of the individual elements and the coherence of the overall package, then it is good to ask the question of whether something is really needed. Naturally, that’s rather easier if staff are changing, but even if not it’s still a helpful exercise in managing your portfolio.

The second is to keep students closely involved in the process. We’ve tried really hard to keep our students included in our conversations through this year – just as we do each year – because they remain our key constituency and we want to make sure that they come with us. That’s meant talking about the range of options we can offer, the impact on supervision of dissertations, the priorities for extra-curricular actions we might organise and the general involvement by them in the life of the department.

In this, we have been very fortunate that our students have engaged so positively and constructively. They’ve shown a very engaged approach that has helped us to produce a revised provision that works better for them while respecting what we can offer. It would be fair to say that their input has been critical in getting through the process as well as we have.

And that’s perhaps the third point: having a community.

One of the advantages of having a relatively small department has always been that we’ve had a high level of communication: a problem for one person has typically been a problem for us all. The strength of resolve of everyone involved – staff and students – to find a way through the review and to make the best of the situation is genuinely impressive. That there was so much support from colleagues across the UK was not only a reflection of our disciplinary solidarity, but also of the good reputation that our particular group has been able to build up over the years.

To give just one example, I found myself talking with some of my colleagues about how we’re going to develop our research activities as a group. To hear such good ideas being shared and acted upon, after what has been a tiring year, was really impressive.

So I’m happy to say that I am part of this department and of how we’ve responded. That is deeply coloured by sadness at the loss of some excellent colleagues, whose contributions have reached into every aspect of our work: we will be the poorer with their departure. However, we will not forget the values and professionalism that they helped to build.


Would You Like Coffee With Your Freshman Year?

Starbucks BaristaIf you’re in a radio market like mine, you get to hear ads for the dozens of online education programs offered by Arizona State University (ASU). ASU also garnered publicity not too long ago for its collaboration with Starbucks that provides the company’s customer service employees with access to an affordable college education.

If you’re a regular reader of this blog, you’re also familiar with my prognostications about the technology-driven transformation of post-secondary education — those depressing posts about the financial unsustainability of small colleges with mediocre reputations, the inevitability of a tiered pricing system that conforms to market demand, and the rise of convenience-based learning.

As of today the stars have aligned: ASU has partnered with edX to offer, according to The New York Times, “an online freshman year that will be available worldwide with no admissions process and full university credit.” Cost? Two hundred dollars per credit hour, payable upon passing each course. The total price for the whole program is about $5,000. 

According to MIT professor Anant Agarwal, the CEO of edX, the full complement of courses for the new Global Freshman Academy is projected to be live within twenty-four months.

Say goodbye to your general education requirements.

More ripping up, more starting again

A bit too much removal of constraint

A few weeks ago, I wrote about what my ideal programme of Politics study might look like. In it, I sought to embed a student-led approach that took them more seamlessly into becoming political actors (in the broad sense).

I got some really useful feedback on the idea which I’ve been turning over ever since. And because I can see some potential here for further development of the idea, I’ll use my opportunity to work through those points now.

The big issue is clearly one of operationalisation. As I’ve argued before, usually my view that one should see practical constraints as an opportunity to try out something different: indeed, I often find that it’s mostly because of those constraints that I try something different. Most of us simply don’t have the time, incentive or inclination to go tearing it all up.

Now, in the spirit of the original idea – to think of what I want, rather than what I must – I should be doing the same for turning the three-stream model into practice.

Ideally, I’d be running those streams alongside each other, so they could reinforce and develop each other, and I’d be letting them run as long as was needed to give students the necessary opportunities to learn. If we were dispensing with academic years, then you might see this becoming a multi-year process, where students ‘develop’ into more and more advanced learning and practice, until they reach some pre-determined set of learning objectives. And part of that would be students’ ability to know that they had reached that level (rather than being told by someone else).

Now, writing that last paragraph has given me mild palpitations (and it’s my idea). To dispense with everything is fun, but it’s also rather disquieting. Not least because it lies quite a long way beyond what we might reasonably encounter. So let’s try something more conventional.

Let’s assume the following. You have academic years and you have to run some kind of summative assessment. Someone is going to ask for a formal structure of a degree programme and will check that you’ve following it. And you don’t get to do anything too radically different from the rest of your university.

In my case, I have to run four 15-credit modules in each of my two semesters, so already I’d have to struggle to fit a three-stream model into that. One option would be to have more ‘exploration’ early on, with more ‘research’ and ‘engagement’ as students developed, but with a constant presence of all three in each semester. Activities might be run that cross-cut modules: a a big activity might provide ample opportunity for all three streams to draw on, and give a natural coherence to the overall package of study.

We also have a lot of joint-honours students, who only take a limited number of our modules each semester, so if we wanted to keep that, we’d need to think about activities that could contain particular strands where those students could take part, without having to become completely involved. So an activity around an election campaign (for example) might include specialist activities of relevance to sociologists (studying ‘the bubble’) or economists (costing manifesto pledges) that would also feed into the work of the single-honours students.

Assessment in these modules would have to be broadly constructed, since the detail of the activities might be largely co-constructed with the students, but if this were embedded at the start of each module (much in the manner of problem-based learning (PBL)), then that needn’t be a major problem, since mutually-acceptable goals and assessment would be set. These might combine both practical activity and self-reflection.

Within activities, there is obviously lots of scope for different approaches, as I noted in the original post. Certainly – as one colleague noted – there’s much in common with a PBL style, but equally there would be no particular need to use one style throughout. Indeed, there would be a strong argument to say that mixing and matching pedagogies would help to ground work and to expose students to a broad range of methods. As we like to say, methods should follow the question, not the other way around.

An example

Let’s try now to put this into effect on the most modest of scales: a module/course. This might be useful, since I imagine that’s what most readers will have to work with. And since it’s that time of the electoral cycle, let’s work on that idea I mentioned earlier of a project on a general election.

Since this would only be one module/course, we’ll assume the students are a bit further along in their studies, so not first-years. And I’ll run it through my second semester, which goes from early February to late May: the British general election is 7 May.

The ‘exploration’ element would be largely implicit, since students would know about electoral systems and political institutions, but would be developed by the ‘research’ and ‘engagement’ streams.

The former would consist of different sub-groups working on aspects of the campaign that they think are of particular interest: identification of party strategies, manifesto coding, media/crisis management, role of non-party actors in shaping debate, etc. This requires a number of different research methodologies, and students could be engaged in one or more of these activities.

The ‘engagement’ side would have two elements. One would be to organise a public event prior to the election, such as a hustings or a town-hall style meeting, where they would facilitate debate. This would side-step being party-political, while still participating in political debate and further embedding their research within a wider social context: indeed, the event itself could be used to provide data for later coding and evaluation.

The second element would be post-election, where students could present their research findings, again to a wider audience: probably this would need to be online, with a website providing speedy analysis of the data collected. Logically, that work would be more locally focused, but the site could also be run throughout the semester as a resource for voters too.

By blending research and engagement, students gain a much more rounded sense of how their study fits into a larger picture, teaching staff get to tap into key political debates with a bunch of young researchers alongside them, and the public get something that might help them participate politically.

And that’s just one module.


Structure and Agency in Quality Enhancement

Sarah’s plan to learn by osmosis wasn’t going well

Lately, I’ve been thinking about what makes a ‘good’ learning experience for students. There are a number of reasons for this, not least of which is the impending visit from the Quality Assurance Agency this autumn.

The question matters because it forms the bedrock of the educational mission of any institution and if we don’t really reflect upon it, then we risk making mistakes or, at the very least, not making the best of the situation we have.

With this in mind, it seems to me that there are at least four elements at play here.

  1. The students. Learning needs learners, so students have to be at the centre of any learning environment. This means we have to play attention to both the intrinsic and extrinsic aspects of learning: building on their desire to learn and providing them with an appropriate motivational architecture to help move them along through their studies.
  2. The teaching staff. For all that we talk about the use of active- and student-led-learning on this blog, that still requires an engaged and committed teaching staff, to design and facilitate those activities. The step back from the passive model does not mean that staff simply fall to the side.
  3. The learning activities. It’s all well and good to have the bodies, but they need to be doing something that enables them to have the opportunity to access the learning objectives that they set out to achieve.
  4. The learning environment. This is perhaps more neglected than the others, but the space in which learning occurs is also consequently. This should be understood in both the narrow (what’s my classroom like?) and the broad (what’s my institution like and what does it claim to achieve?) senses.

Hopefully, some brief reflection can help us see how these four elements come together in our own cases to shape our practice and our efforts to improve it. Likewise, we can see how difficulties in one area can affect the others and require us to take compensatory action.

To take one example, our students here at Surrey have been changing over recent years: the raising of entry requirements has clearly shifted the type of person who comes through our doors. That has had implications for our teaching, since the prior knowledge, the motivation and the anticipation of those new students has had clear impacts on what and how we create learning environments. To a certain extent, our pre-existing focus on active learning absorbed some of this (since these tend to be more openly constructed), but it has still required much thought about the nature and scope of provision.

The difficulty comes largely in predicting what matters.

The list above is an analytical one, rather than a predictive one – it says we should look at these different elements, which all interact with one another.

While that’s true, it is still helpful to engage in that reflection when reviewing curricula to ensure that as much as possible of ‘what matters’ is noted and accounted for.