To Quiz or Not to Quiz, Part 3

Some final thoughts on adding in-class quizzes to my course on economic development:

For six of the nine quizzes administered so far, students answered only half of the questions correctly. Given the results of my survey on students’ study habits, I am increasingly convinced that the problem of transfer is contributing to their poor performance. Perhaps I should create a series of real world-based practice exercises for next year’s iteration of this course. These exercises could be an additional connection to the reading assignments.

Even though each quiz has a maximum of four questions, the quiz-taking eats up a significant amount of classroom time. Perhaps I should impose a time limit. If I put the quizzes online for completion outside of class, students will be able to search for correct answers, which defeats my purpose of testing recall to strengthen memory.

The quizzes have helped me identify what students still don’t know. Reviewing questions in class after grading each quiz might have helped students better understand the concepts that they had been tested on. But the final exam that I created for the course (Part 8 below) will allow me to only indirectly infer whether this occurred. Maybe next year I should repeat some of the same questions across multiple quizzes, or introduce summative exams, to get a better idea of whether students are in fact learning what they are being quizzed about.

Links to the original series on redesigning this course:

Ethically Simulating

This post was inspired by a story in The New York Times: a U.S. Marine discovered his daughters reading a “choose your own adventure” book with a chapter about a battle in Afghanistan in which he actually fought. He thought the book presented a superficial view of war and wrote an editorial on the subject. The book’s publisher decided to stop selling it and halt production of four other similar titles.

The story got me thinking about my own teaching. I often place students in simulated environments that in real life are horrendous — such as genocide, civil war, and natural disasters.* I do this because I believe it does a better job than just reading a text at getting students invested in the subject and developing a less-biased understanding of others. In the past these simulations have even included the digital version of choose your own adventure books.  

Yet I probably don’t pay enough attention to the risk that these exercises can come across as trite games completely divorced from reality. On the one hand, I teach undergraduates who in many cases have lived a materially comfortable life within a psychologically-comfortable bubble. Their world is diametrically opposed to the one that I am hoping they are learning about, and that probably gives them far less of an ability to empathize than I would like. On the other hand, my graduate courses are filled with active and former military personnel. Many of them have direct experience thinking through situations that my simulations attempt to artificially replicate.  

Perhaps I should be asking students “Did this simulation respect reality in a way that contributed to your learning?”

*Inside Disaster: Haiti, the death of which I reported in 2015, has apparently been resurrected via subscription-only access. While its history demonstrates the inherent problems of online simulations, I strongly recommend this product.

Alternatives to Traditional Research Papers

Today we have another guest post by Charity Butcher, Associate Professor of Political Science at Kennesaw State University. She can be reached at cbutche2[at]kennesaw[dot]edu.

Research papers are a common tool used to help students learn about a particular topic. However, students have become accustomed to using information in different ways, and will also be expected to present information differently in their future careers. I therefore decided to give students in my American Foreign Policy course the option of writing a traditional research paper or completing the same research project in a different format – a podcast, video, or poster. Nearly half the students in the class chose one of the alternative formats.

For the assignment, students were asked to choose a current American foreign policy issue, such as U.S. relations with a specific country or a broader foreign policy topic like development aid, human trafficking, climate change, or terrorism. Students first submitted proposals that outlined their topic, included a preliminary bibliography, and identified which format they had chosen. The end product had to describe the foreign policy issue and its importance to the U.S., analyze past U.S. foreign policies on the subject, and recommend future policy. Regardless of the format, students were evaluated on how well they addressed these elements.

Each of the alternative formats had pros and cons. For podcasts, students could include information that was similar in quantity to the traditional research paper. On the other hand, some students first wrote a paper and then read it for the podcast, making some podcasts less dynamic and creative than I had hoped. Overall the podcast option seemed to generate the same effects as a research paper, but added an extra step for students.  

The videos were more dynamic than the podcasts and generally included the same amount of content as a traditional paper. Students were very creative in how they presented information, signaling a bit more thinking than the traditional paper. The downside was that the videos were significantly more time consuming than papers for students to produce. Several students experienced technical problems.

Posters, which students had to present in class, were quite successful. The poster option allowed students to practice their presentation skills, though this occupied class time. It was also more difficult for students to include as much information on a poster as in a paper, though some of this additional information did get communicated in presentations.

Overall, I felt this experiment was successful. In the future, I will eliminate the podcast option and have more specific grading rubrics for each project format. Grades for video and poster formats should incorporate criteria on visual design and presentation delivery. I may also add other presentation options, such as Prezi. I may even add a blogging option!

The Death Spiral

Nearly five years ago I wrote about the dim prospects for Iowa Wesleyan University (formerly College). The school had just announced that it would eliminate half of its academic majors and terminate more than forty percent of its faculty members.

Earlier this month, Iowa Wesleyan’s president announced that it was in danger of shutting down. The university’s trustees decided on November 15 to keep the university operating until at least December 2019. My guess is that their decision just delays the inevitable. According to Iowa Wesleyan’s IRS filings, it suffered from negative net revenue — a deficit — for five of six fiscal years from 2011 to 2016. Operating expenses per full-time equivalent undergraduate increased by more than fifty percent during the same period. Supposedly the university needs $4.6 million in additional revenue to stay open until the end of 2019.

Iowa Wesleyan faces the same problem that many other similarly-sized private colleges and universities face: declining demand coupled with increased operating costs. For many small-enrollment, tuition-dependent higher education institutions, the future is financially unsustainable. 

I’ll be writing more about how to interpret the relationship between a school’s enrollment and operating expenses in the coming months, both here and, possibly, at Inside Higher Ed.

A Basic Mistake

Back in July I wrote about a new writing assignment that I would be inserting into the undergraduate course that I’m now teaching. As is my standard practice, the directions for the assignment included a statement that students should refer to the rubric accompanying the assignment for an indication of how their work would be evaluated. I also instructed students to write about some of the concepts discussed by Tomer Perry in his series on best practices in game design (I put links to the relevant blog posts in the assignment directions).

Here is my mistake: the rubric did not have any criteria specific to the game design principles that students were supposed to address. When grading students’ essays, I didn’t think it was fair of me to penalize them for not writing about something that I had failed to include in the rubric. This was a reminder of a lesson I have learned repeatedly over the years but occasionally forget.

To Quiz or Not to Quiz, Part 2

As promised in my last post, here are the results of the survey of students’ study habits in my course on economic development. Sixteen of seventeen students in the class completed the survey (after repeated reminders); a good response rate although not statistically-significant sample.

When in class, I . . .

  • write notes on a laptop: 25%.
  • write notes on paper by hand: 75%.
  • do not write notes: 0%.

For quizzes, I usually prepare by (choose all that apply):

  • re-reading assigned articles and book chapters: 18.8%.
  • reviewing notes taken during class: 100%.
  • reviewing notes taken when I read assigned texts the first time: 12.5%.
  • doing nothing: 0%.

While I was pleasantly surprised at the number of students who said they take notes — especially handwritten ones — in class, I told the class that highlighting or underlining is an unproductive study technique. It doesn’t engage the motor and language areas of the brain like writing does. Also, when reviewing prior to a quiz or exam, often one has no memory of why one marked a sentence in a text as important. Whether students will heed my advice, I don’t know.

Continue reading

To Quiz or Not to Quiz

Earlier this year I wrote a series of posts on redesigning my course on economic development. As part of the redesign, I added ten in-class quizzes as a way to reinforce students’ learning of important concepts. From my perspective, the quizzes are simple — a maximum of four questions, many of which can be adequately answered with short phrase or sentence. For example:

Why do the rural poor tend to have more children than the urban rich?

However, students’ scores on the first four quizzes have been quite low. I started wondering if bad study habits were causing the terrible quiz performance. Not being able to read students’ minds or observe their behaviors outside of class, I decided to do something unusual: ask them. I created an anonymous online survey with Google Forms. Here are the questions:

When reading assigned texts, I . . . 

  • write notes about important ideas as I’m reading.
  • highlight, underline, or otherwise mark important ideas as I’m reading.
  • do not write notes or mark up text as I’m reading.

When in class, I . . .

  • write notes on a laptop.
  • write notes on paper by hand.
  • do not write notes.

For quizzes, I usually prepare by (choose all that apply):

  • re-reading assigned articles and book chapters.
  • reviewing notes taken during class.
  • reviewing notes taken when I read assigned texts the first time.
  • doing nothing.

I’ll publish the survey results next week.

Links to the original series on redesigning this course:

Interviewing the EU in Brussels

Today we have a guest post by Jamal Shahin and Claske Vos, faculty in European Studies at the University of Amsterdam. Jamal also works at the Institute for University Studies at Vrije Universiteit Brussel. They can be reached at shahin[at]uva[dot]nl and C[dot]Vos[at]uva[dot]nl.

Students in our one-year M.A. programme at the University of Amerstam needed to simultaneously gain knowledge of the EU and develop research skills, all in a four-week ‘skills seminar’ that runs every January. We decided to address this challenge by having students research policy by gathering data
 first-hand from EU policymakers.

We start by inviting the students to consider their ‘burning question’—what they really want to explore—in their MA theses. In the seminar’s second week, they identify methods and theoretical approaches, with a specific focus on the relationship between policy fields and research methods. Students draft deliverables, which are then peer reviewed at this stage. In the fourth week, students learn more about the practice of European policymaking by interacting with EU officials and other ‘experts’ in Brussels. To prepare for this stage, students must first:

  • Arrange the interviews.
  • Create an interview protocol—semi-structured interview guide.
  • Present a literature and policy review in written form.

These tasks are intended to give students a clearer view of the topics they wish to raise with their interviewees. At the end of the entire process, the students are expected to write up a thesis proposal to submit to their thesis supervisor.

From first contact to first meeting

Many students are daunted by the prospect of interviewing experts in fields that they feel that they are only just starting to understand. It therefore takes some effort to assure them that their requests for information will not go unheard. We help the students write the initial emails that request an interview, and provide them with a rough template that describes the etiquette to use when presenting oneself to the potential informant. Our multidisciplinary faculty, which includes anthropologists, historians, political economists, and political scientists, can help the students express themselves to their interviewees. We hold a half-day workshop with the students, in which a range of lecturers from different departments present their own interview experiences.

Bursting the ‘Brussels bubble’

‘Brussels’ is seen by many as a distant place, remote from the realities of daily life, even for students of European policymaking. This course helps students break through this perceived bubble by allowing them to engage directly with the policymakers themselves, rather than studying the policy process only through literature. It also occurs after an intensive eight-week course on European integration theories, and thus helps bring this knowledge ‘to life’—something that for students is frequently an unexpected benefit.

Negatives of Nuance

Related to Simon‘s and Amanda‘s recent posts about failing to get the results one expects, here is a very simple example.

My first writing assignment prompt this semester for my course on economic development and environmental change read as follows:

Purpose of this response: learn about the nature of poverty and perceptions of the poor.

1. Read the rubric below.

2. Read/watch:

  • MRU: Basic Facts of Growth and Development.
  • Easterly, Ch. 1 and Intermezzo, p. 5-19.
  • Emily Badger and Quoc Trung Bui, “In 83 Million Eviction Records, a Sweeping and Intimate New Look at Housing in America,” The New York Times.
  • Hans Rosling, “The Magic Washing Machine,” TEDWomen, December 2010.

Answer the following question:

  • William Easterly writes “When those of us from rich countries look at poor countries today, we see our own past poverty.” (p. 19). Is this an accurate view of poverty outside and inside the USA? Why?

My intent in asking the question — which I thought was obvious from the “statement of purpose” that prefaced it — was to get students to think about the differences between how they think about the poor at home and about the poor in countries that they’ve probably only seen on the news, if at all. In other words, I wanted them to start thinking about why our perceptions of the poor often depend on where the poor are located.

Instead students wrote about wealth and poverty from a historical perspective — that the USA once had a low level of economic development, as measured by GDP or income per capita, and that’s where other countries are today. The USA developed into a high-income, industrialized society, so probably other countries, even the poorest ones, will eventually do the same. There was no real discussion of the nature of poverty in the USA or why it exists.

So I need to a better question, something like “Are the causes of poverty in the USA the same as in other countries? Why?” or “If the USA is an economically-developed country, why does it still have poor people?”

Beyond the Essay: Briefing Memos

Today we have a guest post from Vincent Druliolle, an assistant professor at Universidad Carlos III de Madrid. He can be reached at Vincent[dot]Druliolle[at]gmail[dot]com.

Undergraduates are repeatedly told that what they study is somehow relevant for practice, yet most assignments are structured as academic essays—even though only a handful of them will end up opting for an academic career. A few years ago, I decided that my students should have the opportunity to develop non-academic writing skills, and started assigning a briefing memo about an ongoing conflict.

The briefing memo is indeed a format widely used in government, international organisations, consultancies, and NGOs. However, because of the large range of topics and theoretical perspectives covered by my module and the limited number of teaching weeks, I had to find a way of integrating such an activity into my small-group seminars. I came up with the idea of making the memo a preparatory activity for my in-class simulation on peacebuilding and transitional justice.

The briefing memo differs from the traditional essay in both content and format. It is policy-oriented, because it is aimed at practitioners and decision-makers, and it presents information in a concise and attractive manner. It requires critically analysing source material beyond the standard academic literature, selecting what’s most relevant, and presenting it in a way that can convey the complexities of the conflict analysed.

Most students have never written a memo, but I don’t give them any guidelines. Instead, I ask them to look up examples that they can use as models. I prefer to ask the students to present their memos in class and discuss the difficulties of writing it. The first seminar of the simulation is thus about comparing and learning from the work of one’s fellow classmates. For class discussion, I recommend selecting at least a very good memo, a (very) bad one, and a few with significantly different formats and/or content. The greater the variety of memos, the better. I want the students to learn from each other, so I adopt the role of a facilitator, asking them to explain why they’ve chosen a given format and/or content, and fostering a class discussion about these aspects.

Many students admit that, as I warn them beforehand, it’s difficult at the beginning to figure out how they have to write the memo. Instead of assessing it at this stage, I ask the students to submit a revised version after the simulation that reflects what they’ve learnt from their classmates’ memos. Guidelines about how to write a memo can be provided at this stage or even afterward as part of a debriefing.

While writing the memo is an activity in its own right, in this case it is also a way for the students to acquire the knowledge necessary to participate in the simulation. They learn what information the memo should include because they have to put themselves in the shoes of the actors for whom the memo is written in the first place. In this way, the memo prepares students for the simulation, while the simulation provides an opportunity for them to reflect on the writing of the memo. And for the instructor, memos are quicker (and less boring) to mark than essays.