What are exams for?

For most people, the question posed in the title is either blindingly obvious or completely left-field: exams are to test people, duh.  However, you are most people so let’s try asking the question again: what are exams for?

 I ask for the simple that it’s a question which bothers me every time I encounter an exam.  It’s all the more pressing this semester, as some of my students are sitting an exam that I’ve written. It’s a bog-standard, 2-essay-questions-in-2-hours, closed book exam.

 This is perhaps a key part of my doubt about exams: they end up being a very standardised piece of assessment, relatively unvarying in format and purpose (i.e. getting students to pull together their learning into relatively concise, quick-to-mark answers). One might argue (as people have argued to me, usually when I’m their external examiner) that this predictability of form allows students to refine their technique and maximise their performance.  In brief, their attraction lies as much in their ease of administering as in their pedagogic value.

What particularly troubles me is that exams have very limited real-world utility: when do you ever find yourself stuck in a room with no access to other materials and get told to produce a reflective analysis on a topic that you previously only knew roughly which aspects to prepare beforehand?  Given that we spend as much time as we do asking students to become good managers of information and efficient and effective researchers, why then do we provide them with an assessment form that doesn’t let them demonstrate this?

Obviously, there is a baby/bathwater issue here. If we have an efficient way of assessing (that also properly captures student ability), then we should use it: none of us has unlimited time.  As I’ve discussed before, you can have open book exams, or seen exams (where the student is told what the questions will be some time in advance), or (I suppose) group exams.  These would certainly add something to the mix and encourage some additional skills and abilities.

I do recognise that this is also partly a matter of taste: as a student, I prefered writing essays to exams, even if my performance was as good in the latter.  But still we need to come back to the original question, albeit in a different way: what do we want to assess and what’s the best way to do that?

More Thoughts on Modular Course Architecture

My spring semester is over. Undergraduate students have completed an anonymous evaluation of my comparative politics course, in which I experimented with modular architecture. In this course, students chose one of five different themes to focus on for the semester — political identity, democratization, revolution, genocide, or globalization. The themes were cross-indexed by geographic region; for example, in a particular week, all students’ readings were about Latin America. At the beginning of the semester, all students read journal articles on all five themes as an introduction.

A few non-scientific conclusions about what students wrote in their evaluations:

A few students — despite regular reminders — refuse to acknowledge the importance of  the “plan ahead” concept. Yes, this book is 300 pages long. No, it’s not something you can read the night before the essay on it is due. That’s why I give you a syllabus at the beginning of the semester.

Some choice is good, but too much is bad. The reading assignments for the genocide theme were all books. I let each student in this theme choose three books to read, which meant that a given student wasn’t obligated to read anything for some of the geographic regions. A few students said they felt lost in class discussions as a result.

Groups needed to have members that were all doing the same thing. Throughout the semester, students had to do group presentations that consisted of close-readings of assigned texts. Students did not select the themes in equal numbers, so some of the groups that I created contained students who had chosen different themes. Students in these mixed groups said this hampered collaboration — one or two members of the group would contribute nothing. My expectation was that students would teach content to each other while they were putting their presentations together. So that didn’t work out. In the future I will either need to drop this kind of collaborative task or figure out a way to ensure relatively equally-sized groups, each containing students who have chosen the same theme.

Class discussions helped students see connections between different historical events and apply theoretical concepts to new situations.  One of my personal goals as a teacher is to facilitate students’ ability to  integrate knowledge, so I would like to develop more formal ways of doing this — graded writing assignments, if possible — in the future.

Sticking it to the man

We’ve already reached the last week of teaching this semester, so it’s been time for me to round up my feedback from students.  In particular, I’ve been keen to find out what they thought of both my use of sticks to randomly select seminar presentations and the open book multiple-choice exam.  As before, I started from an ABC exercise to generate some data to open the conversation.

In both cases, the students were very happy to discuss the pros and cons, which suggests that they have at least engaged at that level.

With the sticks, there was a considerable amount of unhappiness about their use. “It’s unfair”, “they make people not attend”, “if people haven’t prepared very well, then we just sit around doing nothing” were some of the comments.  At the same time, several people had made positive comments about the random nature of the system and the way it had made them prepare more than they otherwise might have done.

Reading more widely across the comments, a couple of people noted that seminar leaders hadn’t been consistent in ejecting those who hadn’t prepared and my feeling from the discussion was that this might have been a key part of it all: if there was strong confidence about others preparing, then students would themselves prepare.  This is certainly one point that I will be taking into the next iteration.

On the MCQ, the picture was less clear.  Some pointed to all the assessment being test-based, when they would prefer at least one essay.  Others liked the MCQ, but not the open-book part of it: “it’s not fair that I got a high mark” was one telling comment.  Others still obviously enjoyed the experience, although how much that is a reflection of the high marks I cannot tell.  The argument about essay-based assessment is a fair one and fits my general tendencies in this, but for now my inclination is to try it for another year, perhaps without the use of computers or smart-phones, or even more conventionally closed-book.

Beyond this, some of the comments seemed to suggest that these (first year) students hadn’t all grasped that they are not expected to be active learners, rather than people to be taught.  They enjoyed my lecturing style and the (metaphorical) ‘love’, but wanted more detailed powerpoint slides and discussion focused on revision/assessment topics, as well as more direction on what they needed to know.

In all of this there is the central difficulty of balance, between giving students want they want and giving them want they need.  As I said in the class, I’m always happy taking on feedback, because it’s the best way for me to square that particular circle.

How Soon Is Now? The End of the University As We Know It

The university as we know it is headed for extinction.

Imagine the U.S. higher education system in the early 1960s – universities were awash in government money because of the Cold War; enrollments were surging because of the GI bill and the baby boom. The paradigm of undergraduate education was the four-year, full-time, residential college experience, best suited for unmarried 18-22 year olds who could delay entry into the workforce. The university was the Emerald City in the Land of Oz – a fortress of knowledge where truth was revealed to humble supplicants.

Compare that with the current educational landscape. According to the U.S. Department of Education, approximately forty percent of undergraduate  students are older than 24. About half are classified as financially independent of parents. Forty percent attend college on a part-time basis. A huge percentage of students simply don’t have the time for the four-year, full-time, residential college experience that was the norm in 1960.

Simultaneously a bachelor’s degree has gotten a lot more expensive. In inflation-adjusted dollars, the average price of a four-year college education is 2.5 times what it was in 1980. During the same period, real wages for most Americans have been flat. People in the USA now need a college education just to maintain the same standard of living enjoyed by their parents and grandparents, but the cost of that education keeps going up. So more students go into greater amounts of debt to finance their educations.

As the traditional college model has become less convenient and more expensive, the opposite has happened to information. When I grew up, newspapers printed the news on paper, radio programs were broadcast over the air, and there were three TV channels. Then cable, satellite, and the internet completely disrupted the media industry. Today anyone can access almost any kind of news or entertainment at any time, and, perhaps even more importantly, anyone can create it – much of the content is user-generated. In many ways this transformation resembles Gutenberg’s invention of the movable-type printing press, which broke the monopoly on information that had existed in Europe for centuries.

The same is now happening to higher ed.

The Emerald City is being replaced by an information ecosystem that resembles a coral reef. There’s no moat, no wall, no gatekeeper controlling how the information is accessed or interpreted. It is open source and the architecture is scalable.

According to Wael Fakharany, Google’s regional director for Egypt and North Africa, worldwide there are 4 billion Google searches and 4 billion online video playbacks every day. Every second, someone uploads an hour of video to YouTube. These webpages, videos, and the ability to search them are free.

At this point you are probably thinking that 1) the internet is an unorganized and unreliable mess, and 2) undergraduate students still need the traditional university to organize, evaluate, and deliver educational content effectively.


People interested in learning have been migrating from free but unorganized educational online resources – the webpages that turn up in a Google search and the videos uploaded onto YouTube – to free organized information for several years. Now the movement is toward educational systems – housed at least in part online and frequently open-source – that offer convenient, affordable, and credentialed content mastery.

The simplest example of free organized information is Wikipedia, which started 2001. Wikipedia’s content is 100 percent user-generated.

A step up in terms of quality is MIT’s OpenCourseWare, which began in 2002 and currently has material from over 2,000 courses, such as classical mechanics.

Khan Academy, which began in 2006, now has over 3,000 video tutorials. Khan Academy’s motto is to provide a “free world-class education to anyone anywhere.”

The nonprofit educational foundation TedEd, a spinoff of TED, went live last month.

People can now use this free online content to learn anything, anywhere, anytime. There’s just one problem: how do you demonstrate what you’ve learned to others? If you walk into a job interview and say “I learned chemistry from Khan Academy,” the person on the other side of the table will simply shout “Next!” You need some sort of recognized, accepted credential signifying that you learned what you say you’ve learned.

Historically credentialing has been the function of the course credit and diploma offered by the traditional university. It has enabled universities to maintain an expensive monopoly on higher education, despite inroads by lower-cost but less prestigious for-profit institutions.

Until now.

Students who take the equivalent of an online self-paced correspondence course from the non-profit Saylor Foundation (motto: “education should be free”) and receive a certificate. So what, right? Students can then take an exam administered by the for-profit StraighterLine; if they pass they receive American Council on Education (ACE) credit, which then can be transferred to other educational institutions.

Still not impressed? This month, Saylor and StraighterLine began a partnership with Northern Virginia Community College and George Mason University to make college more accessible and lower the cost of completing a degree. The plan is to allow students to begin their college education at any of these four institutions; prospective students will know upfront the benefits and costs of each starting point.

A different method of unbundling undergraduate education was pioneered at Stanford by Sebastian Thrun and Peter Norvig last fall. Thrun and Norvig taught a massive open online course on artificial intelligence that enrolled 160,000 students. Students were ranked according to performance in the course and received a certificate upon completion.

This year, Norvig and Thrun started Udacity, which currently offers six free computer science courses and enrolls a total of 200,000 students. Udacity’s plan is to offer certification to students at an affordable cost.

A similar venture, MITxoffered its first free online course, on electrical engineering, this year. MIT has now partnered with Harvard to create edX, a $60 million nonprofit joint venture, to offer free online instruction worldwide. Like Udacity, edX will offer certificates demonstrating successful completion of courses for a small fee.

At this point, you might be thinking that these developments aren’t relevant, because you teach political science, or history, or English, not computer science. These innovations will affect you, if they haven’t already, in three ways.

First, higher education is headed toward a system of price discrimination that resembles a Starbucks menu. Elite universities with established brands will continue to draw students who have the necessary time and money for the traditional full-time, four-year campus experience. The mid- and low-tier brick-and-mortar schools will be forced, due to the increasing convenience and cost-pressure of open source alternatives, to redesign their curriculums or die. They will need to abandon uniform pricing for course credit so that they can hit as many price points as possible, and they will need to go after more undergraduate students who are older, employed full-time, and unwilling to come to campus three days a week between the hours of 9:00 a.m. and 3:00 p.m.

Second, the transformation of the higher education market will force universities to better demonstrate that the product they are selling justifies the price at which they are selling it. Each item on the educational menu will need to clearly reflect “you pay for what you get and you get what you pay for” because of what open online courses will do to credentialing. Grade inflation is rampant in American higher education, and the massively scaled competition offered by open online courses can produce a credentialing system that is far better than what brick-and-mortar universities – even many of the most prestigious ones – are offering. For example, let’s say that Cal Tech gets 10,000 applications a year but admits only 1,000. You’re admitted, you enroll, and eventually you graduate at the top of your class. In the end, though, you are still 1 of only 10,000. Now let’s say that instead of going to Cal Tech, you take a computer science course from Udacity, or a whole set of them, along with 500,000 other people. You end up being the top student. Best of 500,000 is likely to be a lot more meaningful to an employer than best of 10,000, and you didn’t have to pay $200,000 in tuition, room, and board to prove it. Think what that’s going to do to the universities that aren’t Cal Tech but that (for now) charge $200,000 for a bachelor’s degree.

Third, instructors will have to pay greater attention to whether their teaching creates value for the student and the university. Massive open online courses are going to generate huge amounts of data on teaching and learning. They’ll be able to demonstrate the effectiveness of particular teaching methods much more easily than the compartmentalized classroom environment. The pedagogies that are developed and tested online will filter into the physical classroom, and anyone who is not interested in continuous improvement in their teaching will lose students to the people who are.

Perhaps more importantly, course content will move out of the physical classroom and onto the internet. Open source initiatives like MIT’s OpenCourseWare demonstrate that course content is the least valuable educational component that universities have to offer, so it’s simply given away for free. The more valuable forms of learning that cannot be easily replicated online – research, project design and execution, problem solving, peer-to-peer learning – will take over the classroom. It will be more important for students to figure out how to get from Point A to Point B than for an instructor to tell them that Point B is better than Point A. The lecture at the podium will cease to exist.

As it moves online, educational content will be created and delivered in a distributed manner. Both students and instructors will become accustomed to feeding it forward – they will locate and analyze what others have done, adapt it for their own purposes, and then make what they’ve created available for others to use in their own learning.

Instructors who are unwilling or unable to engage with content or students in this manner will be at a disadvantage. Unfortunately most PhD programs currently do not provide training in this pedagogy to the graduate students who will be entering the teaching profession.

Disclaimer: I have no financial interest in any of the entities discussed above.Thanks to Sally Gomaa, members of the PeckNet, and Chana Motobu for their helpful comments. Any mistakes are the fault of my cats, who like to sit on my keyboard.

More Lego-based learning

This weekend I’ve been learning about learning from a young boy of my acquaintance.  Upon arrival at a weekend of wedding celebrations for his aunt, he discovered that while he had brought his recently-acquired Lego X-Wing starfighter – dismantled into its 400 or so individual bricks – he had left the instructions back home.

After some flumping about, he was encouraged to try building it from memory, since he has already built it and dismantled it (with instructions) about ten times already.

2 hours later, he was finished.

I talked to him about this, since I was intrigued as to his approach, not least since he had clearly not been specifically prepared for such a task (a common issue when learning).

He explained that some parts were very easy/memorable and didn’t require any thought as to their construction.  Secondly, he noted that there were sub-assemblies he could work: blocks of bricks that naturally belonged together and which could be dealt with, then put to one side until they could be joined on.

However, the main part of the model didn’t work this way, since it was an integrated design that needed to be done in a particular order. Here was where I found the most interesting insight, namely that he had thought about the intent of the finished piece: what was it that each piece contributed to the overall design?

As always, this made me think of how our students acquire and use their learning.  the key difference here is that students don’t get asked to exactly reproduce an output (or rather, they shouldn’t be).  But they do have to engage in novel exercises that might be out of their direct experience, extrapolating from what they have previously done.  Just as this boy progressively moved away from the instruction booklet as the weekend progressed on his iterations of building, so students develop their confidence in using knowledge and skills through use.  And as instructors, we need to think about how we can facilitate this.


The importance of information literacy can be difficult for undergraduates to grasp. For most of them, all information is created equal and it comes from Google. Now Google is helping to educate students out of these habits. The website includes sample lesson plans and videos for instructors who want to integrate information  literacy training into face-to-face or online courses.

Instructors can also use A Google A Day to test students search skills — it’s fast, fun, but not necessarily easy.

Do I mind about mind maps?

I’ve always had an ambivalent relationships with mind maps*.  Despite the frequent suggestion from various quarters about how great they are, lots of freeware online and their obvious value in helping to visualise relationships within a subject, I have tended to find them rather problematic.

This has been for two main reasons.  On the one hand, it tends to impose a hierarchy of organisation, working out from a central node, which tends to make the incorporation of cross-cutting themes at best tricky and at worst a complete mess.  On the other, I find that my way of understanding a subject tends to be more linear: I need a starting point to build an argument and a position.  As result, mind maps haven’t figured very much in either my teaching or my research.

However, that’s been changing a bit during this semester’s teaching.  One of my classes is a Masters module on EU policies, which I teach to a small group of about half a dozen students.  This has allowed me to get us all around a table to talk with either other, rather than a more formal arrangement.  In the second half of the module, we’ve been looking at individual policy areas and this is where the mind maps have shown their value.

This is my effort for the EU’s Common Security & Defence Policy and is my copy, essentially representing my notes for the class.  As we talked about the policy area, I re-created much of this diagram with additional points, working on an A1 flip-chart sheet.  as you can see, the map allows me to piece together some ideas and gives some focus to sub-sections within the topic.

From the students’ point of view, this has helped in capturing a relatively complex field and (particularly) in getting to understand how they might tackle the second phase.  This is a student-led session where they propose changes to the policy area in order to improve its operation.  The mind map lets them get a better sense of what the problems are and how potential changes might impact.

I’m still not totally sold on mind maps, but I can see that sometimes they are a very helpful tool.

    • This is exactly the kind of sentence that gives academics a bad name with the rest of the population.

What In The World Is Critical Thinking?

The Likert item teaching evaluation instruments completed by our undergraduate college students usually contain something along the lines of “The course improved my critical thinking skills.” This 2011 Religion Dispatches column by Nathan Schneider, about why the world needs religious studies, points out that the phrase “critical thinking” is for the most part an empty cliche for students, policymakers, and the public at large. Schneider argues that if we think the liberal arts should be an integral part of higher education, we need to specify that our courses teach students how to:

  1. evaluate truth claims
  2. identify value
  3. apply diverse methods to problem solving
  4. create and disseminate persuasive narratives.

And then we need to gather the data to demonstrate that our students in fact acquire these skills because of what we do in the classroom.

As a first step in this process, I’ve been replacing all references to critical thinking in my syllabi with the above phrases.  When I do a close reading of the syllabus with students on the first day of class, that’s what I’ll be focusing on.

Asking a Good Question

My experiment in using modular architecture in course design is coming to a close. I don’t have the result of student evaluations yet, but a few students have commented that they’ve enjoyed the structure of the course.

One part of the course hasn’t worked out as well as I’d hoped. My syllabus contains the following instructions:

Five times during the course of the semester you will be asked to write a brief reflection (equivalent to one half to a full page of double-spaced 11 or 12 point text) and raise a critical question (or questions) stemming from a previous class discussion.  You may want to clarify a particular point made in class, critique a particular point, wonder about the implications of a particular idea, or consider the relationship between one author’s writing and another.  In short, these questions can go in the direction of your choosing, but they should be clear, concise, and original. Discussion reflections and questions are submitted online and questions will be used as a basis for conversation in class.

The above task was intended to 1) get students to write and think more about the reading material, 2) promote peer discussion both online and in class, and 3) get students to create exam questions so that exams became more formative learning exercises.

I had the most success with (1). Students did frequently reflect upon what they had read, and occasionally they connected the ideas of particular authors to what I or fellow students had mentioned in class.

To my surprise students did not engage in online discussion about the merits of each other’s questions, even though they knew I was looking for questions to add to the exams. Instead I had to break students into groups in class, with each group selecting a favorite question that had been posted online to discuss. Each group then reported what it had discussed to the rest of the class.

I would have liked more back and forth between the students, whether online or in the classroom. I would also have liked to see better formulated questions – I was hoping the students themselves would point out the lousy questions, but this didn’t happen.

If anyone has any advice on how I can improve this process, I’m all ears.