As I mentioned earlier this summer, I’m collaborating with two students to build a simulation for my introduction to comparative politics course. I’ll continue to blog about it (and I’ve invited them to write a post or two) throughout the fall semester.
One question that came up was not about the content of the simulation, but about how (and whether) to assess its effectiveness. A unique opportunity arose as I will be teaching the same course to two sections in my program. The students who enrolled were randomly divided into two classes of 17. As an advocate of active learning techniques and a quantitative scholar, I was tempted to use one of the two classes as a control group and only run the simulation in one section (the treatment). Then I could present the results at the 2016 APSA Teaching and Learning conference. What could be better? It’s the holy grail of pedagogical research.
During one of our planning meetings, my collaborators and I discussed this idea. We were all struck, however, by the lingering question: if we think simulations are such a great teaching technique, why only give half the students this opportunity? Does the value of demonstrating effectiveness (if that is, indeed, what the control vs. treatment experience would have shown) trump providing all students the enriching experience?
Beyond that, the logistics of teaching two classes the same for most of the semester, but doing something different for the final three days (when the simulation will run) is burdensome. The classes would be imbalanced not simply because one had the simulation and the other didn’t, but because the control class would have three days of material or activities that the treatment class wouldn’t. What would I do in those three days that would be fair to both classes, while not contaminating the results of the assessment comparison?
Finally, the students in the program, although in different classes, will still live in the same dorm and know each other. No doubt, they will compare their class experiences. If the simulation is great, the students in the control class may feel left out (and, although unlikely, if the simulation is a flop, those students will feel as if they got the short end of the straw).
After some discussion, we settled on running the simulation in both classes. Now I must brainstorm other ways to assess its effectiveness.
It’s the old Harvard Med School story. A world famous surgeon is discussing a new technique that he says saved many more lives among the patients he used it on.
From the back of the room, and uncertain voice asks, “Did you have any controls?”
The surgeon sneers and says, “Do you mean did I divide the patients into two groups randomly and only use the technique on one half?”
The voice in the back says, “Well … yes, that is what I meant.”
The surgeon smashes his fist on to the lecture and roars, “That would have condemned half of the patients to death!”
Out of the vast silence, the uncertain voice says, “Which half?”
Yes, that’s it. Don’t think; do the experiment.