To Quiz or Not to Quiz, Part 3

Some final thoughts on adding in-class quizzes to my course on economic development:

For six of the nine quizzes administered so far, students answered only half of the questions correctly. Given the results of my survey on students’ study habits, I am increasingly convinced that the problem of transfer is contributing to their poor performance. Perhaps I should create a series of real world-based practice exercises for next year’s iteration of this course. These exercises could be an additional connection to the reading assignments.

Even though each quiz has a maximum of four questions, the quiz-taking eats up a significant amount of classroom time. Perhaps I should impose a time limit. If I put the quizzes online for completion outside of class, students will be able to search for correct answers, which defeats my purpose of testing recall to strengthen memory.

The quizzes have helped me identify what students still don’t know. Reviewing questions in class after grading each quiz might have helped students better understand the concepts that they had been tested on. But the final exam that I created for the course (Part 8 below) will allow me to only indirectly infer whether this occurred. Maybe next year I should repeat some of the same questions across multiple quizzes, or introduce summative exams, to get a better idea of whether students are in fact learning what they are being quizzed about.

Links to the original series on redesigning this course:

(Trans)formative Assessment in Teaching

Today I’m attending ISA’s inaugural Innovative Pedagogy Conference in St. Louis.  Victor and I are doing a workshop on using games and simulations to teach political violence, showcasing activities like Survive or Die!, Prisoner’s DilemmaModel Diplomacy, identity exercise, and others.  But I’m most interested in reflecting on the session offered by Matthew Krain and Kent Kille of the College of Wooster on Assessment and Effectiveness in Active Teaching in International Studies. Their framework for formative assessment (that can, in fact, be transformative) is very helpful as an overall approach to teaching.

Continue reading

The price of failure

via GIPHY

After last week’s class discussion about participation, I decided to run an exercise that made it really easy to show the marginal benefit of preparation.

I told students to prepare for a meeting about putting together an agenda for another negotiation, and gave them all specific roles, plus some rules of procedure.

(For those who are looking for Brexit sims, this was a Council working group, putting together an agenda for the Commission to take to the UK to discuss the Political Declaration).

Because it was about formulating an agenda, I hoped that students would see they didn’t need to get too deeply into substantive positions, as long as they could frame the general areas to be covered.

Plus, but giving clear roles and rules, I incentivised everyone to push out their own draft agendas prior to the meeting. In so doing, I hoped they’d see that even a small amount of preparation could have big effects.

Um

Obviously, it didn’t turn out that way.

Continue reading

To Quiz or Not to Quiz, Part 2

As promised in my last post, here are the results of the survey of students’ study habits in my course on economic development. Sixteen of seventeen students in the class completed the survey (after repeated reminders); a good response rate although not statistically-significant sample.

When in class, I . . .

  • write notes on a laptop: 25%.
  • write notes on paper by hand: 75%.
  • do not write notes: 0%.

For quizzes, I usually prepare by (choose all that apply):

  • re-reading assigned articles and book chapters: 18.8%.
  • reviewing notes taken during class: 100%.
  • reviewing notes taken when I read assigned texts the first time: 12.5%.
  • doing nothing: 0%.

While I was pleasantly surprised at the number of students who said they take notes — especially handwritten ones — in class, I told the class that highlighting or underlining is an unproductive study technique. It doesn’t engage the motor and language areas of the brain like writing does. Also, when reviewing prior to a quiz or exam, often one has no memory of why one marked a sentence in a text as important. Whether students will heed my advice, I don’t know.

Continue reading

To Quiz or Not to Quiz

Earlier this year I wrote a series of posts on redesigning my course on economic development. As part of the redesign, I added ten in-class quizzes as a way to reinforce students’ learning of important concepts. From my perspective, the quizzes are simple — a maximum of four questions, many of which can be adequately answered with short phrase or sentence. For example:

Why do the rural poor tend to have more children than the urban rich?

However, students’ scores on the first four quizzes have been quite low. I started wondering if bad study habits were causing the terrible quiz performance. Not being able to read students’ minds or observe their behaviors outside of class, I decided to do something unusual: ask them. I created an anonymous online survey with Google Forms. Here are the questions:

When reading assigned texts, I . . . 

  • write notes about important ideas as I’m reading.
  • highlight, underline, or otherwise mark important ideas as I’m reading.
  • do not write notes or mark up text as I’m reading.

When in class, I . . .

  • write notes on a laptop.
  • write notes on paper by hand.
  • do not write notes.

For quizzes, I usually prepare by (choose all that apply):

  • re-reading assigned articles and book chapters.
  • reviewing notes taken during class.
  • reviewing notes taken when I read assigned texts the first time.
  • doing nothing.

I’ll publish the survey results next week.

Links to the original series on redesigning this course:

Badges and recognizing success

Badges are not exactly a new concept, but like many ideas that come from games, they are still not widely known in the world of higher education and learning.  As many of us know from playing sports or participating in Scouts-style programs as children, badges are physical marks of achievement given to those that show competency at a particular skill or for doing well in a competition. In video games like World of Warcraft, meeting a particular goal is called an ‘achievement’.  Each achievement you earn flashes across the public chat, allowing for recognition, and some of them come with special items or titles in the game.  Defeat the Lich King, and you not only get to bask in the glory of victory, but you earn the title ‘Kingslayer’. While typically meant as a form of visible recognition for an achievement, earning achievements or badges can motivate behavior and can be seen as credentials in their own right.

Motivation, recognition, and credentialing skills–sounds like what we want to do in the classroom, right?

Assessing Student Preparation

This is follow-up to a previous guest post by Joel Moore on the benefits of assessing students’ preparation for class discussion rather than their participation in it. Students are asked to rate themselves on the amount of assigned reading they have completed prior to class, and this forms the basis of their preparation grade.

He has created a web app that simplifies the process. The app is available at https://joeldmoore.com/apps/preparation/.

A video that discusses how to use the app is at https://www.youtube.com/watch?v=SQdK1FwycKw.

Transparent Teaching in Action: Sometimes you aren’t as good as you think you are

In the last few months I’ve been interested in transparent teaching. Basically, transparency is a commitment to all of your students to be transparent about your expectations for a class, particularly when it comes to course assignments. Transparent teaching requires being clear and precise about 3 things in an assignment:

  1. Purpose–explain to the students WHY you are asking them to do this assignment.  Don’t just assume they know how it connects to the material.  Talk to them about the skills and knowledge they will gain and how that is relevant to the course, major, program, or other aspects of their lives.
  2. Tasks–be explicit about the tasks students must follow to complete the assignment.  Sometimes we tell our students to ‘write a 5 page essay’ but don’t realize that many of our students might not know what we mean by an essay–it means different things in different fields, and not all of your students will have taken a polisci class before. Of the three areas, this is where we tend to be the most detailed in our assignments, but even here we can probably be more clear about exactly what we want than we area.
  3. Criteria for Success–the students need to know how their work will be evaluated to determine whether they have successfully completed the assignment or not.  This might mean a rubric, or just a list of what you are going to be looking at.  But one of the most important elements–and the one I messed up on–is it provide examples of successful work.  This might be student work, or something written by a professional working in the field. Want students to write a strong literature review? Show them what a strong one looks like, and talk to them about what makes it strong.  Contrast it with unsuccessful or weak work.  Better yet, give them the examples and the rubric, and have THEM score the work, so they understand how the criteria is applied.

Yes, all of that takes time.  But we owe it to our students to give them every chance to achieve success in the work we assign them.  If we think it is valuable for them to do this work, then we need to give them the detail and time it takes so that success is entirely in their hands.  Plus, doing this has wider benefits.

Research by the Transparency in Learning and Teaching (TILT) team at the University of Nevada in Las Vegas has shown that increasing transparency in our assignments doesn’t just impact their work on those assignments. It leads to big gains in student confidence and sense of belonging in college, particularly for disadvantaged populations who come to college less prepared.  At UNLV, they saw a 20% increase in retention for these students who participated in classes where just two assignments were transformed to become more transparent. I led a project at my university last spring on transparency, and while I’m still evaluating the data, those who participated indicated they found the training sessions and transformation process very useful.  If you want to attend a training on TILT, review their extensive resources, or read the ongoing research, head to their Resources page.

So here I am, someone who has trained on transparent teaching, conducted multiple trainings myself, and doing research in the area.  I’ve also used specifications grading in the past, which is in the transparent teaching wheelhouse.  I’m a pro, right?

Yeah, right.  Every time I think I’ve ‘got’ something when it comes to teaching, I end up learning how much more I have to learn.

I’m teaching a new course this semester called Sex, Marriage, and Violence, and I’m running it largely as a seminar.  Students have to write weekly papers on the readings and pose discussion questions that form the basis for class.  In the syllabus I included a purpose statement for these papers as well as a rubric, and the instructions (I thought) were clear: I wanted a full APA style bibliographic entry for each article or chapter, a 1-2 sentence summary of their main claims or findings, a 500-1000 word analysis of the themes, debates, gaps or issues raised by the readings for the week, and 1-2 discussion questions.

The first papers showed up on Monday, and all but one of them failed to follow this format.  Only a few did the bibliographic entries or summaries; some put the summaries in the main body of the paper (something I explicitly said not to do); only a handful gave discussion questions at the end; and a few wrote essentially annotated bibliographies, with no effort to compare or synthesize the readings.

A couple of students messing up is probably their fault; almost everyone messing up is clearly mine.

The good news is that this is fixable.  Because this is an assignment that starts early and repeats weekly, I was able to catch the problem in week 2.  This is an advantage of using smaller stakes assignments throughout the class–there is less harm to the student if they are completely off base in their efforts, and you have a chance to teach them how to improve. 

To fix this, I recognized a key mistake on my part: while I had provided a rubric, I had not provided an example of successful work.  So I drew up a 2 page outline of a paper.  I didn’t write an entire paper myself.  But the outline showed them the formatting I wanted to see in the paper, such as showing them that I wanted the bibliographic entires and summaries BEFORE the paper proper, and the discussion questions AFTER, not embedded in the text.  And I wrote an example of a thesis statement based on the readings for Week 2 along with a single paragraph showing them how to analyze one of the readings according to that theme.  Drawing up this document took me about an hour, but the students SO appreciated it.  They now had a visual aid to see what I wanted with examples, not just a list of requirements and criteria.  I also allowed them to resubmit their paper by the next class, and throughout the course, will ask students to keep anonymized versions of their work so I will have student-written examples to show future classes.

All of this goes to show that even when you are trying to be transparent, sometimes you might not be able to anticipate what kind of information or models the students need to really understand what you want them to do. When that happens, it is typically OUR fault as instructors, not theirs as students. Now, some students will get it wrong no matter how clear you are, but there are definitely times when we are wrong when we think we are being clear.  I’ve sometimes thought that the ‘provide examples of successful work, and show students why it is successful’ is optional, but on reflection I think this is perhaps the most important thing we can do*. This was an important wake-up call for me that even when I think I’m doing a good job, I can still get it wrong. 

*There are those who question providing examples of successful work lest students simply copy it.  I completely understand the critique, but i think with creativity on our part, this can be overcome. Often I am demonstrating formatting and technique, not content, so as long as you change the content out, there’s little risk of copying.  My go-to example is to make arguments about why cats are better pets than dogs, and I can illustrate the expected structure of an essay, a bibliographic entry or citation, a thesis statement, and using evidence to defend a claim using this very non-political example.

Another point here is that providing a structural template for students who aren’t strong writers is super useful to them.  I’m fine with them imitating my structure–I’m teaching them how to write a strong paper.  I encourage students who have more advanced writing skills to branch away from the suggested structure in my outline/template to find their own style.  

Changing a Course on Development, Part 8

I want my final exam for this course to be its pièce de résistance — a vehicle for students to demonstrate how well they can apply their knowledge about the relationships between economics and the environment. I also want the application of knowledge to happen in an authentic, real-world context, where writing has a clearly-defined role, audience, purpose, and format. So here is the exam:

A Plan for Louisiana’s Future

Role

You are the Director of the Office of Planning and Budget for the State of Louisiana.

Audience

The governor of Louisiana

Purpose

Recommend to the governor whether Louisiana should either:

  1. Raise taxes to build the southern part of the state to a 10,000 year flood standard, or
  2. Stop all public infrastructure spending in areas unprotected by existing levees.

These are your only policy options. Write a 2-3 page rationale for choosing one of them. Discuss why your choice is economically best for the state. For evidence in support of your rationale, refer to relevant course readings and Continue reading

Additional Snippets from Comparative Politics

A few more quick thoughts about my comparative politics course, which just ended . . .

On the last day of class, I distributed pieces of paper and asked students to write down what they thought were the two best and two worst assigned readings, and to include brief explanations of their choices. Nine students were in the room — out a class of ten. Despite such a small sample, I will go out on a limb and draw some conclusions, all of which relate to transparency:

  • I should preface each writing assignment with a very brief overview of how assigned readings relate to the topic at hand, because students don’t automatically know this. For the question “Does Chinese culture promote authoritarianism?” the class read an interview with Singapore’s former leader Lee Kuan Yew. Two of the students didn’t understand that Yew, being of Chinese ancestry like many other Singaporeans, was discussing his views on Chinese cultural norms.
  • I should also explicitly inform students that readings are written in different styles for different audiences, which means that some of the readings will be more difficult and less enjoyable to read than others. For example, in the section of the course on revolution, I assign Theda Skocpol’s 1976 article “France, Russia, China: A Structural Analysis of Social Revolutions.” Two students said this article was a least favorite reading, because of its style. Oddly, in my opinion, a few other students complained that the long-form journalism found in some of The New York Times articles was hard to understand.
  • Students have difficulty placing unfamiliar historical events in chronological order and parsing out how what happened before might have affected what happened after (the “history just happens” mentality). For an assignment about Iran, students felt confused about the relationship between the 1979 revolution, popular support for the post-revolutionary Islamic Republic, and more recent anti-government protests. While I find it easy to connect the dots, students don’t — for them, context can be confusing. So for some assignments I need to narrow down the reading list.

After students finished identifying readings they liked or disliked, I asked them if they would have preferred formal mid-term and final exams, or regular quizzes, as an alternative to some of the writing assignments. I was surprised that they said no, since I had been under the impression that they used test scores to reassure themselves of their performance. One senior said that he didn’t remember a thing from test-based courses he had taken, while he remembered a lot from writing-based courses, and therefore he regarded the latter as much more beneficial. Two others said that, as studies have shown, they and their friends promptly forgot whatever facts they dumped into short-term memory the night before an exam, so they regarded exams as anxiety-producing events irrelevant to their learning. If next year I use quizzes as a stick to improve attendance in class, they will probably need to be the “write a paragraph that answers the question” type.