Teaching the Value of Replicability

Guest contributor Markus Gastinger is at the Technical University of Dresden. He can be contacted through http://markus-gastinger.eu/ and is willing to share his material upon request.

Markus Gastinger

In the winter term of 2015/2016, I was teaching a course on international organizations (IOs). The course was attended by 25 Master students and supported by a lecture. Since not all students had a prior degree in Political Science, we began by reviewing three major IR theories – realism, neoliberal institutionalism, and social constructivism. We had a fair mix of readings for all three but realism was underrepresented because of its state-centered ontology. We discussed constructivist texts with true zeal but they did not – and I write this with great deference and sincere admiration – lend themselves to (quick) lessons in replicability. This cannot be said of rational-choice contributions. After we had discussed the Principal–Agent (PA) model, we read an important contribution by Liesbet Hooghe and Gary Marks (2015) published in The Review of International Organizations. This article struck a chord with students since its large-N approach to studying delegation (the transfer of authority to an international secretariat) and pooling (the transfer of authority to a collective member-state body) gave them an overview of 72 IOs.

Around Christmas the students approached me saying that, although we discussed texts thoroughly, they felt they were not able to apply concepts. Consequently, in the week before Christmas we discussed replicability and its value to allow others to build on one’s findings. The controversy over the Data Access and Research Transparency (DA-RT) initiative was in full swing, giving us the opportunity to link our debate to real-world events. Hooghe and Marks (2015) is a distinctly positive example of transparency. We looked at their supplementary files when we discussed the article in one of the previous sessions but this time I asked them to actually replicate the study for the World Trade Organization (WTO), International Monetary fund (IMF), and International Bank for Reconstruction and Development (IBRD). Hooghe and Marks have nine indicators to measure delegation, for example whether the international secretariat is involved in the accession of new members, drafting the budget or reaching policy decisions. Students had to use these indicators to code the founding documents in a binary fashion. To provide a contrast with this quantitative approach I added central PA concepts drawn from Hawkins et al. (2006), asking them to grasp similar notions qualitatively. Since university regulations did not allow me to give credit for this assignment participation was entirely voluntary.

Frankly, I was skeptical they would plough through the legalese of the founding documents. But, oh boy, was I in for a surprise! We had spirited debates before but this was taking it to a new level. All were eager to share their findings with me. This did not necessarily make my life easier. When in the past students would, somewhat passively, accept this or that definition or finding, now they became very critical. They realized that behind concepts, numbers, figures or tables were always humans. While Hooghe and Marks is without a doubt an excellent piece of research that deserves attention, the intricacies inherent in any form of conceptualization, operationalization, and measurement introduce a certain element of fuzziness (e.g., where to draw the line between 0 and 1?) and, perhaps at times, discretion (e.g., why these nine indicators and not others?) that one can probably only truly appreciate after a replication exercise like this.

Delegation, pooling and PA tools all became much clearer with this assignment, which gave students first-hand insight into (re-)coding. One thing that I need to change next time is to identify the international secretariat before the assignment. Perceiving the need to find high levels of delegation, many erroneously looked for delegation to the executive board (IMF), executive directors (IBRD) or General Council (WTO) rather than the managing director, president, and director-general. This frustrated many since they felt this should have been pointed out in advance. Nevertheless, not only did this exercise give them a glimpse of the complexities behind political-science research and the value of replicability. Somewhat tongue-in-cheek, it also helped train the next generation of student coders.

References

Hawkins, D. G., Lake, D. A., Nielson, D. L. and Tierney, M. J. (2006) ‘Delegation under anarchy: states, international organizations, and principal-agent theory’, in D. G. Hawkins, D. A. Lake, D. L. Nielson, and M. J. Tierney (eds). Delegation and Agency in International Organizations. Cambridge: Cambridge University Press, pp. 3–38.

Hooghe, L. and Marks, G. (2015) ‘Delegation and pooling in international organizations’, The Review of International Organizations 10(3): 305–328.

 

Leave a Reply