A Lesson Learned About Team Research Projects

Looking at student performance in the 2020-2021 academic year, I see evidence that team research projects due at the end of the semester can’t be scaffolded solely around individually-graded assignments completed throughout the semester. For example, in my Middle East politics course, each student shared four individually-completed assignments with their teammates for use in their team’s historical timeline. In my research methods course, there were ten individual assignments that teammates were supposed to share with each other as drafts of sections of team research reports. While this approach does decrease free riding and encourage collaboration, it apparently does not ensure high quality research in the final product. Four of the five timelines that teams created in the Middle East course lacked mention of significant events. None of the four teams in the research methods course collected information from coffee farmers, processors, or distributors in Central America, despite my instructions to do so, nor did the final reports resemble the industry exemplars I had provided.

It seems that in students’ minds, my formative assessment of their individual work is totally unconnected to the summative assessment of their collaborative work. I probably need to break the team project into discrete, graded chunks, with each chunk layered on top of some of the individual assignments. Teams can use the feedback they receive on each successive chunk of the project to improve the quality of the final product.

Improving Simulation Efficacy With a Scaffolded Final Exam

A follow-up to my post in April about making exams exercises in active learning:

From the very beginning of my teaching career, I’ve emphasized, or at least tried to emphasize, the importance of being able to construct evidence-based arguments. My exams are almost always intended to evaluate students’ proficiency at this task. As I mention in the post linked to above, the final exam for my comparative politics course in Spring 2020 included the stock phrase of:

reference course readings to support your argument.

For the final exam in Spring 2021, I substituted:

support your argument with 1) information from the Gerkhania server on Discord, and 2) cited references to at least two red and two green journal articles listed in the syllabus.

Explicitly requiring the citation of four articles that students were nominally already familiar with from previous assignments resulted in greater use of scholarly evidence in exam essays than had typically occurred in the past. Students sometimes didn’t use these sources in ways that actually supported their arguments, but in these cases I could tell that at least an attempt had been made.

However, to my surprise, not a single student referred to specific player behavior during the simulation. That is not how students read “information from the Gerkhania server on Discord.” Instead, they summarized the simulation’s outcome or, worse, repeated the general background information on Gerkhania that I had provided before the simulation began. So, for 2022, the exam prompt should probably include something like:

support your argument with 1) examples of specific actions made by players during the Gerkhania simulation, and 2) cited references to at least two red and two green journal articles listed in the syllabus.

This is all well and good, because my main purpose for the final exam is summative assessment of learning. But I also want the final exam to help me gauge whether the Gerkhania simulation contributed effectively to this learning. While the first part of my potential exam prompt gets at this question indirectly, I think more is needed. So I have been thinking about “scaffolding” the final exam around the simulation.

I typically run Gerkhania over three sessions. It occurred to me that I could assign something like the following after each session:

Which theoretical perspective best explains players’ behavior in today’s Gerkhania session? Why? Refer to specific player actions and Course Reading X in your response.

These assignments would be short pieces of writing, easy for students to complete and for me to grade. They would allow students to practice for the final exam, and they would function as a contemporaneous reflective through-briefing rather than just a post-hoc debriefing. And I would be able to observe whether students’ ability to construct evidence-based arguments about the simulation improved over time.

From the Back Office

A few updates related to maintaining this blog, per Simon’s recent post about institutionalizing one’s digital footprint:

The web host for ALPS is upgrading its server software to PHP version 7.4 in a few weeks. Supposedly this change is compatible with the blog’s current WordPress layout and will not introduce any errors. Fingers crossed, given what happened a little more than a year ago.

A few months ago, as part of an effort to distribute the blog’s content to a wider audience, I had to hire a programmer to disable the default WordPress RSS feed and create a new feed:

https://activelearningps.com/?call_custom_simple_rss=1

So if you use the RSS feed instead of being notified of new posts by email, and ALPS suddenly disappeared from your RSS reader, that’s why.

While the above information might make me sound like a tech guru, I’m not. Part of maintaining this website is figuring out what I can learn on short notice and when to seek help for what I can’t.

A tip of the hat to our long-running sponsor, Soomo Learning. We couldn’t keep the lights on without Soomo’s support. They are good people with a great product. Check them out.

Last, but not least, if all goes according to plan, my favorite colleague and most trusted adviser returns to the USA later today. After six months apart, it will be nice living with my wife again.

Students’ Experiences With Technology During the Pandemic

The nonprofit association EDUCAUSE has released two reports on a study conducted in Fall 2020 about students’ pandemic experiences with:

  • Connectivity and technology.
  • Learning with technology.

Both reports can be accessed for free here.

What for me was the most telling part of the second report:

“students learning experiences were undermined in myriad ways by poor decisions in the delivery and management of courses. On the pedagogical side, students complained of long lectures with massive slides decks . . . assignments with little scaffolding or connections to learning outcomes . . . and generally trying to replicate face-to-face experiences in online learning environments.” (italics mine) 

Discord With Gerkhania

In my comparative politics course this past semester, I ran my usual Gerkhania simulation on Discord as an experiment. Discord is a free social media platform that Amanda has discussed previously. It was a positive experience, for the following reasons:

I had never used Discord before, yet it was very easy to figure out. Discord’s design is very intuitive and setting up the simulation was very simple. Students also found Discord easy to learn.

Students interacted more with each other than they did last year when I used Webex, despite a similarly small class. Webex does not allow for spontaneous communication between participants except for one-to-one chat messages. When building the Discord server, I granted students access to different communication channels according to their roles in the simulation. For example, a student representing an ethnic Khan who practiced the Montian religion had access to channels devoted to each group and could automatically message other Khans or Montians at any time. As server host, I could observe and participate in these conversations in real time.

Discord permits text, voice, and video communication. I deliberately chose not to use its videoconferencing capability and none of the students used it either. We communicated with each other solely through text messages. I believe this enhanced rather than degraded the experience in comparison to Webex — no black boxes instead of faces, and no interrupted video or audio because of low-bandwidth internet connections. A user interface that facilitates text communication also means Discord is suitable for running a simulation like Gerkhania asynchronously rather synchronously, something that isn’t realistic with video-based platforms.

My use of Discord also meant that students automatically had a complete record of the simulation’s events that they could reference for the final exam. I did not have to take any additional steps, like create and share a recording, for the class to have a history of what had transpired.

My Latest Iteration of Teammate Evaluations

A brief note about end-of-semester teammate evaluations:

I again used Google Forms to distribute a survey for students to evaluate each other’s contributions to team projects, but I changed how I calculated this component of the course grade. Each student had twelve points to distribute across all team members, including themselves. The more valuable a person’s contribution to the team project, the more points that person was supposed to get. People who made equivalent contributions could have been awarded the same number of points, and if a person was judged as having made no contribution at all, he or she could have been given zero points.

When the Google Form closed, I computed an average number of points received for each student. I then divided this mean score by twelve and multiplied it by fifty (the teammate evaluation was worth 50 out of 1,000 points in the course). I used this formula because teams were larger than in previous semesters, and I assumed a few members of each team would do the heavy lifting with the rest doing little or no work. If the resulting number was fifty or higher, a student earned the full fifty points toward his or her course grade. If the result was below ten, the student earned nothing. For any number in between, I rounded to the nearest ten.

This past semester, I had a total of thirty-seven undergraduate students in two courses. Only thirty completed the evaluation. Four of the thirty completed the survey incorrectly — the scores they distributed across team members did not sum to twelve. I deleted their responses, as I had specified in email and in the Google Form’s directions.

In sum, approximately thirty percent of my students did not perform a simple task that could have benefited their own course grades.

As I speculated at the end of the Fall 2020 semester, I was able to label the teammate evaluation as being worth zero points on Canvas. Maybe that partially explains why no students have (so far) complained about this portion of the course grade.

Judson College: Dead

Yesterday Judson College’s board of trustees voted to close the institution.

I briefly profiled Judson’s financial woes back in February.

My advice? If you are working at a college with fewer than 1,000 students and an endowment of less than a billion dollars, find employment elsewhere.

Can Final Exams Be Active Learning Exercises?

More on the recent topic of active learning strategies that don’t involve simulations . . . but first some meta-babble on how I understand the concept:

To be considered “active learning,” a task should require learners to apply prior knowledge in a novel way or within an unfamiliar context — what the cognitive scientists refer to as transfer. The activity should, in some fashion, resemble lived or expected experience, because people learn more if they see what they are learning as relevant (a feature referred to as authenticity). The activity should also be organized so that learning occurs efficiently. Finally, the learner should be cognizant of the learning process, which means reflecting on what has been learned and why.

In the social sciences and humanities, exams, especially final exams, are rarely regarded as opportunities for active learning. Perhaps they should be.

Continue reading “Can Final Exams Be Active Learning Exercises?”

The Online Field Research Project

To pick up the gauntlet metaphorically thrown down by Amanda last week, here is the first of what will probably be a series of posts on my experience teaching an introduction to research methods course online this semester. When I last taught this course two years ago, I used Amanda’s Best Breakfast in Town project. Given the constraints imposed by the coronavirus pandemic, sending students into restaurants simply wasn’t an option this time around. Yet I still wanted students to experience the trials and tribulations of real-world field research. I decided create a new research project on specialty coffees from Central America, with teams investigating coffee from Costa Rica, El Salvador, Guatemala, and Honduras, respectively. To increase the authenticity of the project, students are responsible for designing a survey (replete with a pilot test and my coaching to try to avoid problems like sampling bias), conducting remote interviews with the people who produce and sell these coffees, analyzing the resulting primary source quantitative and qualitative data, and communicating their conclusions in an industry-style report.

Continue reading “The Online Field Research Project”

Reality Check

As a response to the situation described in my last post, I created an in-class exercise for my comparative politics course — this worksheet:

1. Write the main thesis of these articles by changing each article’s title into a declarative sentence containing “because,” “causes,” “is caused by,” etc.:

Ronald Inglehart and Christian Welzel, “How Development Leads to Democracy: What We Know About Modernization,” Foreign Affairs 88, 2 (March/April 2009): 33-48.

Alfred Stepan, “Brazil’s Decentralized Federalism: Bringing Government Closer to the Citizens?” Daedalus 129, 2 (Spring 2000): 145-169.

Larry Diamond, “Why Are There No Arab Democracies?” Journal of Democracy 21, 1 (January 2010): 93-112.

Javier Corrales, “Authoritarian Survival: Why Maduro Hasn’t Fallen,” Journal of Democracy 31, 3 (July 2020): 39-53.

Scott Mainwaring, “The Crisis of Representation in the Andes,” Journal of Democracy 17, 3 (July 2006): 13-27.

2. Fill in the blank cells in the table below with information from Larry Diamond, “Why Are There No Arab Democracies?”:

3. Given the above, what causes the value of the dependent variable? How do you know this? Is this a Most Different Systems Design or a Most Similar Systems Design?

I gave students 10 minutes to work on these questions individually, followed by 10 minutes in breakout rooms with teammates to discuss their answers. Afterward, I reviewed the lesson by asking students to state what they wrote for each thesis or blank table cell. This occupied the remaining 30 minutes of class. While I don’t know what students thought of this exercise, it’s something I can use in the future either in the same way or as part of an exam.