Graded Online Discussions: Update

Switchboard OperatorsBack in August I complained about the lack of student discussion in two sparsely-enrolled online graduate courses. I’m now past the mid-point in two other such courses, and I thought I’d post an update.

When teaching online, I frame each week’s discussion around a question that relates to the week’s reading assignments. Now that I explicitly grade the discussions using a feature of the Canvas LMS, I see comments that are much more on point and thoughtful, so to me there is no reason to revert back to the old format in which I really just tallied the number of comments made by each student.

In contrast to the summer courses, I’m getting a lot more discussion. I believe class size is the driving factor, not the season. In my current courses, there are still some students who post comments rarely or not at all, but in absolute terms there are more people in each course who are willing to carry on a conversation — which leads to more conversation.

In the first week of these courses, I did notice what appeared to be an upper bound on class size for productive discussion. Students who commented later in the week repeated ideas that many of their classmates had posted earlier. There is little chance for originality after reading fifteen responses to the same question. I solved this problem by switching on another Canvas feature, “users must post before seeing replies.”

Cleaning up other people’s mess (in a good way)

3175714614_321ac67ede_bA few weeks ago, I wrote about simulating the Greek crisis. I suggested then that one issue in doing this was the difficulty of carrying things over from year to year in the classroom: students change, curricula change, you never quite know whether it’s still going to be relevant, etc.

As Amanda rightly pointed out in an email to me some time later, you can perfectly well do it, with a bit of thought. So it’s with that bit of thought that I am now doing it.

Last year I created my first online asynchronous simulation for the INOTLES project in which I participate. As you’ll see from the post, it’s a simplified recreation of the East European situation, with a friendly (if ponderous) EU-like structure on one side and a confident (if worryingly so) Russia-like country on the other.

I played this with my students too, with the upshot that the ‘Russians’ produced a surprising success in sealing a deal with the ‘East Europeans’ (largely over a misunderstanding, but let’s not pretend that doesn’t happen in real life too). I put the simulation back on the shelf, mused on what had happened and then basically forgot about it.

Until Amanda’s email. There’s no reason why this year’s students can’t pick up where their predecessors left off.

It’s a fictional scenario, with all the requisite information provided. Since it allows for a wide-ranging set of actions, there is no obvious end-point or stable equilibrium. Indeed, one might imagine that some students might take the opportunity to revise the actions of the past, just because they can. Certainly, given the rather devil-may-care approach to a second round of the Hobbes games in class yesterday, that looks like a rather likely outcome.

I’ll quote Amanda at some length here:

Whenever we do a simulation, it tends to be a new run of an old game–how neat would it be to have the simulation just continue, with students acting as the newly appointed representative for that country and having to work with old agreements produced by students who are no longer in power?  I find the idea really interesting, not only for the sense of realism it brings to ongoing negotiations, but also for the real-world skill of having to step into a job vacated by someone else and having to figure out what the prior office holder did and how to incorporate their decisions into your own

Amanda’s last point is perhaps the crucial one: we all have to pick up other people’s stuff and deal with it – it’s a basic stable of professional life – so getting to experience that is a useful opportunity for personal development.

Indeed, in this game the original conceit that it opened with no particular situation is clearly unrealistic, so we’ll learn about path dependency directly.

Amanda’s one concern was about record-keeping: how to capture what had happened, so that we can pick it up again. Well, I’ll admit that this isn’t a big issue in this case. The final agreement reached ran to a full four lines of hand-written text and there was nothing else to share. I’m hoping that this time around we’ll have clearer sight of the next year, so that paper-trails can be left, with all the joys that brings.

As usual, this is all new territory for me, so I’ll be reporting back as we progress.

Perils of Small (Online) Courses, Part 2

How I see my course design
How I see my course design

Two weeks ago I wrote about the lack of discussion in two online graduate courses with a combined enrollment of eleven students. These two courses ended yesterday and I’m now taking a break from grading final exams. Discussion decreased to zero in the last week of one of the courses and remained paltry in the other.

I created an ungraded, anonymous survey to try to find out what students thought was most useful about these courses. Perhaps how I think about course design, which includes weekly discussion, is very different from how students think about it. However, only two people responded to the survey, so I have to guess both about the causes of the problem and about potential remedies.

None of these remedies are very attractive to me. I could increase the weight of discussion participation in the final grade to something like 20 percent. But as I wrote previously, my online courses are populated primarily by adult students who have competing demands on their time — such as family and full-time careers. I do not want to penalize people too severely if writing about the course readings is all they have time for, especially since it’s my belief that students learn more from this than from participating in discussions with classmates who often have erroneous or biased points of view.

A second option is to require that each student submit a certain number of responses to other students’ discussion comments each week. But with only a half dozen or so students in a course, I suspect they will quickly just be repeating what others have already posted. 

How students might see my course design
How students might see my course design

A third possibility is to alter the discussion medium itself by requiring video rather than text posts.  Video comments were actually suggested at the blended learning conference that I recently attended as a way of enhancing online discussion, and so I inserted brief introductory videos into discussions beginning with the fourth week of class. But there are major drawbacks to this technique, too. Videos aren’t captioned, which contradicts universal design principles. Listening to comments made on video takes longer than reading text when the instructor is grading the quality of the content. And some students will invariably be incapable of producing video of sufficient quality or in a usable format.

The last option is to just abandon online discussions altogether — one less thing for me to grade — when a small number of students enroll in a course. Given students’ lack of engagement with online discussion this year, they must not have found it very important to their learning or to their grades, so eliminating it might not have much of a negative effect.

The Perils of Small (Online) Classes

Shrinking ManA few months ago I wrote about a few of the effects of small class sizes in my undergraduate courses. I’m experiencing related problems in my online graduate courses, where per-class enrollments have dropped to only about a half dozen students.

With so few students, there is no actual conversation in weekly discussions. I fully understand and accept the fact that several factors probably diminish student contributions to discussions — these courses are online, asynchronous, and convenience for students is paramount. However, people did respond to each others’ discussion comments in previous semesters when enrollments were higher.

This is the first time that I’ve explicitly tied discussion posts to a rubric — at least two posts per week, with the first before midnight on Wednesdays, with cited examples from the reading assignments, to earn full credit — rather than vaguely categorize them as evidence of course participation. But making my assessment of students’ contributions to discussions more transparent should encourage rather than discourage posts, if it has any effect at all.

In sum, students are occasionally acknowledging the existence of their classmates but decide to forego interacting with them. They might be reading each others’ discussion posts, but there is no way for me to tell if this is in fact happening; if they are, they aren’t writing responses. Overall it seems unlikely that much social learning is occurring.

Back to the Future with Blended Courses

I recently returned from the Online Learning Consortium’s conference on blended learning. Blended, or hybrid, means a course in which lecture content has been moved online, and less-frequent classroom sessions focus on higher-order tasks of application, evaluation, or synthesis.

Here is the advice that veterans of blended course design gave at the conference:

  • Set student expectations in advance. Students who are new to blended courses frequently conclude that they are a bad combination of the online and face-to-face worlds. It’s up to instructors to frame the experience as one that provides greater access to and more effective interaction with faculty. Pitching the course as an experiment is probably the worst message to send.
  • Online content and face-to-face exercises must correspond to but not duplicate each other. Students’ classroom participation in team- or project-based activities, for example, needs to align with the key concepts of the online content so that both sides of the course unfold in a coherently-scheduled, mutually-reinforcing manner. A frequent method of assessment that prevents non-proficient students from progressing through the content is highly useful in this regard. If online replicates what happens in the classroom, or if they are not integrated with each other, students will either stop engaging with the former or stop being physically present in the latter. 
  • Students need to understand that “online time” does not replace “homework time.” They will still need to devote significant effort outside of class to research, writing, or the completion of problem sets. This message can be highlighted as part of the orientation to using online content that students will need at the beginning of the semester.
  • Conversely, instructors need to be careful not to overwhelm students with material in excess of what students would encounter in the course’s traditional version. 
  • Online video should be in 5-10 minute pieces with Goldilocks-style assessment exercises after each piece — something not too easy nor too difficult. This fosters students’ engagement with the content by giving them the feeling that they’re being fairly challenged. If the assessments are perceived as too difficult or as irrelevant busy work, student motivation to access the content will decrease.
  • When producing video, don’t be afraid to be a real human. Students are not looking for a Taylor Swift-level of production value.
  • Use replicable tools, methods, and content to drive down the financial and emotional costs of creating additional blended courses in the future.

Grading Discussion in Online Courses

Cat ErrorIf enrollment holds steady, on June 29 I will start teaching two seven-week online graduate courses.* I’ve been teaching these courses every summer for several years, and I’ve decided to experiment this summer with a different system for grading student discussions.

I incorporate student discussion into all my courses, whether they are on campus or online, because I believe it fosters student engagement. But–yet again–discussion in these two courses last year demonstrated that there is often a difference between my beliefs about what students should do and how they decide to achieve whatever objectives they have set for themselves.

The shift was also prompted by the adoption of a different instructional tool. When I began teaching these courses, my university used Blackboard as its course management system. Anyone who has used Blackboard knows that it lacks an intuitive user interface and requires that both students and instructors click through innumerable screens. I created this rubric for class discussion, but there was no way to easily link it to what students were writing. Also the rubric was much too complicated to use to evaluate every discussion post by every student. My assessment of discussion defaulted to digging into the student analytics feature after the mini-semester had ended, to weigh the total number of a student’s posts against a scale I had created. Students got little direct feedback from me on how well they were performing in this component of the course while it was still running.

Last year a few students did not participate at all in the weekly discussions. Because of how I structure my courses, they were able to exercise other options and still perform well in terms of their final grades. But their absence from the discussions meant that their peers were not learning from them and they were not learning from their peers. And it looked to me that the lack of transparency in how I evaluated discussion made this outcome more likely.

This time around the courses will be delivered via Canvas instead of Blackboard. Canvas allows the instructor to create interactive rubrics that can be linked to specific assignments or posts in a discussion. The instructor clicks on the rubric’s boxes and the resulting grade is generated. Students see how their work will be assessed without having to click through a myriad of webpages, and they get immediate feedback from the instructor. 

So I created this new rubric, simpler than the old one but still containing the criteria that I think are most important for peer learning in a professional environment, for grading each student’s discussion posts on a week-by-week basis.** I’ll let you know how it works.

*The courses are the politics of the Middle East and comparative political development, part of an M.A. program in international relations. If you’re interested in acquiring some transferable graduate credit hours, learning about a new subject, or learning how to design and teach online course on a compressed schedule, get in touch–you don’t need to be admitted to the degree program to  enroll in either course.

**My wife/colleague showed me how to do this.

Poverty Games, Part 7: Against All Odds

Against All Odds is an online game hosted by the Office of the United Nations High Commissioner for Refugees (UNHCR) that lets players experience what life is like for a political refugee. The game has twelve parts, divided into three sections: War and Conflict, where you play through levels that require you to give up all your beliefs and identity markers in a military police interrogation (or you are thrown in jail indefinitely), choosing which of your many possessions to take with you when you leave (including choosing whether or not to take your dog), and navigating out of the country without being seen; Border Country, which focuses on trying to get asylum and figuring out who to trust; and A New Life, where you have to get a job and go shopping in a climate of suspicion and in some cases, hostility.

There are some great things about this game. Each level only takes a few minutes to play; the entire game can be finished in 30 minutes. The level one initial interrogation is interesting, because you cannot proceed in the game without agreeing to some pretty harsh statements, such as ‘I give up my faith’ or expressing a willingness to give up your language. Students will probably struggle with agreeing to these statements, which can lead to interesting conversations about life in an oppressive environment. Several of the mini-games are fun to play, such as the one where you have to navigate out of town without being seen. You can access any of the twelve levels at any time, which makes it easier to focus on a particular lesson with the students, and each level can be replayed as many times as necessary without penalty. UNHCR also provides a series of web facts, with stories of people who lived through the levels in the game, as well as teaching resources related to the game, although much like the game itself the target audience is younger than college.

There are some downsides. First, some of the levels are a bit too high-handed, such as the ones in the A New Life segment where you have to go from apartment to apartment and learn about your neighbors’ prejudices. Its also frustrating that a single incorrect answer prevents you from directly moving onward in the game (you can always go back to the start screen and choose a later level, but its an inconvenient extra step). For example, one mini-game involves correctly identifying the origin of various inventions like the bicycle, chocolate, gum ball machines, and insulin. A single incorrect answer prevents you from moving on. In one way this is an asset, as replaying it helps you learn, but it’s still cumbersome. Also, the game ends rather abruptly and anti-climatically.

Overall, though, Against all Odds is a pretty neat way to teach students about the plight of refugees, focusing not only on the original oppression that drives refugees out of their home country, but also on the difficulties they face in seeking asylum and building a new life. I would recommend assigning it either as homework the night before a lesson on refugees, or with a small class and access to a computer lab, having them play through a level or two as a class exercise, followed by discussion.  This would also work very, very well in an online, hybrid or blended learning environment.

Previous Posts in this series

Part 1: Ayiti, the Cost of Life

Part 2: 3rd World Farmer

Part 3: Free Rice

Part 4: Spent

Part 5: Inequality Monopoly

Part 6: Papers, Please!

There goes the neighbourhood

Who can you trust these days?

Over the past few months, I’ve been playing an online, asynchronous game with a couple of different groups.

One of these is the INOTLES trainers that I’m training in the use of simulations, while the other is my final year undergraduate students here at Surrey.

As we come towards the end of the INOTLES cycle, I thought I’d finally share my game with you (Neighbourhood Game) and would talk about how it worked.

The game is essentially a simple model of the EU’s Eastern relations, with a three country Union, a big unhelpful country that likes to exert its influence, and a couple of smaller states in-between. In the attached version, I’ve put two players per country, but that can easily be expanded.

The gameplay is very open, with lots of options for activity, coupled to a long period in which to play (using a weekly cycle of posting on a forum).

The aim is to get students thinking about collective action problems in an international context, as well as to cope with the vagaries of online discussion: the INOTLES version has been running with players from six countries.

So how’d it go?

In a word, fitfully.

Online games require a hook, to keep people involved and active. My ability to do this with the INOTLES group was less than with my students (who I see every week for class), but in both cases the level of engagement has been less than with comparable face-to-face exercises. In short, the ability to coerce action is very limited.

That in turn highlights the importance of the game leader in motivating players: if you’re not pushing, then who’s going to do it for you? Again, the online nature of the game means that it’s easy to forget to prod people, and this is something that I’d say needs careful thought.

A second issue seems to be linked to the openness of the game play. Because players can do pretty much anything from writing a terse communique to launching world war 3, I’ve noted a certain hesitancy about doing anything. This ‘jam choice overload‘ problem is well-known in psychology, but raises an interesting problem for us. Too much choice might be inhibiting, but it also reflects the real world, so we might want players to feel inhibited. As always, this will depends much on what you aim to achieve.

Thirdly, while this is a fictional situation, it is also obviously close to the real-world (I took most of the data from real countries). This adds a different dimension to the game, as people apply what they know of that real-world into their actions. Thus Novy Putonova acts rather like Russia, Bigistan like Germany and the Squashed Republic like Ukraine. To be more accurate, people acted like they thought those countries act like. This offers lots of opportunity to get into a discussion about how we understand the real world: are Russians really that sneaky, to take one obvious example? Either way, it opens a door to discussion of the substantive material.

On the level of skills, there is also a lot to think about. How did people work in their teams? How did they deal with each new development? How much did they try to take control of what was happening? How did they cope with some groups being very passive/silent? Again, in all of this, the large range of possible actions meant that there was also a big question about why they chose to do what they did, and not something else?

Next steps

This game has been a trial for me (in at least one sense of the word). It’s been my first effort in this type of game and, as always, I’m not totally happy with it.

On the plus side, players have played, and it’s shown that one can model an international system with some quite simple elements. Feedback to date from both groups has been positive and – importantly – I can see how I would change things in future.

On the down side, engagement has been relatively low (compared to face-to-face) and my input has been more than planned. The inhibition to take more drastic action in either game (they’ve largely been polite and pretty constructive) means that I don’t know how the more radical options might play out.

With my students, I now plan to use our final session before the Christmas break to play the game in class for a couple of hours, to connect it more strongly to the rest of the module, and to let me see how that changes the interactions.

If you’d like to use the game, please do – I’ve popped it up on my other website already. If you’d like to feedback on how it works for you, then I’d also love to hear about that.

Do we still need face-to-face interactions?

Alfred wasn’t sure that the ticker tape was properly conveying the emojis

Yesterday prompted this question from a variety of different directions, but it feeds into a bigger question about the changing nature of Higher Education, especially in light of MOOCs. Face-to-face, physically-co-located meetings are relatively resource- and labour-intensive, so can we get the same benefit without them?

First prompt was the third in a series of webinars that I’ve been involved with for our INOTLES project. Using Abode Connect, we brought together educators in 4 time zones and 7 countries for a ninety minute chat about assessment and feedback. And it worked really well. The mixture of video, audio and text debate produced a rich discussion, from which I think everyone got something. Indeed, precisely because it was one in a series, we were able to focus on the matter in hand, rather than feel – as we have when meeting physically – that we need to cram in everything we can.

Second prompt was my continuing uncertainty about whether I will be joining my fine ALPS-blog colleagues (and Victor) at APSA’s TLC in January: they’ve heard, I’ve not.*

TLC was of course the occasion that first brought us together, and I’ve tried to go every other year, so that we can spend some time catching up and generating new ideas. By necessity, our group is one that interacts almost exclusively online, so we obviously can cope without the meeting, but it’s still good to see each other.

Third prompt was my first serious use of the online marking and feedback system that I have introduced to to our Faculty this year. Quite aside from the technical aspects of this, I’ve also come to realise that it further changes the interaction with students, since it invites you to channel much more feedback through the online system: no handing back of essays in class, with an opportunity for a quick chat.

So, in three very different ways, I have been thinking about the role of face-to-face interactions, both with students and with colleagues.

Now, you might expect that – given my work on simulations – I’d be unreservedly in favour of face-to-face, because of the richness of that mode of interaction. And you’d be right to an extent.

In all three of the prompts I’ve just mentioned, face-to-face modes can bring something that is probably impossible to achieve otherwise. The scope of deep engagement, the building of soft elements of relationships, the unverbal cueing – none of that works in the same way elsewhere (if at all).

But at the same time, I can see a huge value in online interaction: indeed, without it, this blog and the INOTLES project wouldn’t work at all. And the process-tracing and transparency of the online marking/feedback system makes for a more manageable experience than its predecessors (transition costs notwithstanding).

In short, it’s that old chestnut: it’s not better or worse, just different.

Personally and professionally, we need to get students exposed to different kinds of interaction, since they will not always be able to choose what situation they find themselves in.

Seen as such, the important factor is to develop everyone’s reflection about the nature, benefits and limitations of those interactions; and that includes us too. It’s not only useful as a skill in of itself, but it’s also useful in making sure that we all get the most of how we deal with other people.


* – If someone from the TLC committee is reading, I’d really like to go, so let me know.

Online asynchronous simulations – challenges and opportunities

For the past month I’ve been running my first online, asynchronous simulation, as part of the INOTLES project. We’re now coming into the final stretch of the game, so I’m thinking about how it went.

The game itself has a set of states who have to find a modus vivendi between themselves. They look a lot like the EU, some East European places and Russia, but with various things changed, so we don’t just reproduce those places and their relations. Players are given a state, some basic stats and interests and then are left to it. Weekly cycles require everyone to post positions and/or actions by Friday lunch, and if I need to intervene, then I’ll post on Monday morning.

One option I didn’t model…

The practical experience has been rather salutary for me.

Beforehand, I knew that getting participation was going to be the real issue and so it’s proved. The closest participant is about 200 miles away and the furthest is more like 1000 miles distant: teams for states were deliberately mixed up, so anyone playing has had to invest quite some time, just to get their team active. It would be fair to say that not all teams managed this.

I’ve not pushed very hard on this, partly because I have little scope so to do, but also because I’m running this as part of a training trainers exercise, so I want the players to think about how they are going to handle similar situations themselves. It’s a bit jujitsu, but there’s enough there for it to work.

The second big issue has been the asynchronisity/asynchronousness/not playing all at the same time. A week might make sense from the perspective of busy people trying to sort out positions, but it means that in four weeks we’ve not moved very far, or very fast. That reinforces the problems of engagement and participation: without a pressing problem, who’s going to feel a urgent need to react?

And this feeds into a final issue, the very open-ended nature of the game itself.

I’ve kept away from IR-type games, because it seemed like there was a lot I was less comfortable with, plus the notion of trying to capture all that international actors can do to each other was a bit daunting, especially in a made-up scenario. However, colleagues in the project had expressed interest in a sim about the European neighbourhood, so in I plunged.

The paradox seems to me to be that by saying to players that they can do pretty much anything, they end up doing little. Perhaps if I’d limited it to diplomatic exchanges then that might have focused minds some more, but that doesn’t necessarily help in building understanding of the complex interplay of factors. Indeed, I almost feel like using the real-world case would have generated more buy-in, even if it would come at a price of heightened emotional factors.

In short, it’s not easy, building games, especially when you’re trying something new, regardless of how much you’ve done it before.

Once I finish the sim, I’ll debrief the players and use that to inform some redesigning (which is also why I’ve not posted any materials yet). In addition, I’m also getting my students here at Surrey to play the game, all through to Christmas, so I’ve got some scope to try out a couple of ideas on them.

By treating a sim as being in a state of permanent beta, I can live with the uncertainty, and I can plan for it. In part that’s possible because I’m as interested in the process of negotiation and of sims as I am in the substance of the game itself, but it requires a bit of fore-thought and a willingness to adapt on the hoof.