A Field Experiment in Campaigns and Elections

Today’s guest post comes from Dr. Jeffrey Broxmeyer from the University of Toledo!

---

In the Fall of 2024, I introduced a field experiment project into my Campaigns and Elections course. Students set up a basic experiment that followed campaign communications across two congressional races during the 2024 elections in the United States. During the semester, they collected original data, analyzed findings, and presented them to the class.

Goals

The purpose of this experiment was to have students analyze patterns in campaign communication to better understand electoral dynamics. My teaching objective, however, was to demonstrate how social science thinking differs from punditry, journalism, or personal viewpoints.

Specifically, I wanted students to approach the 2024 election cycle as political scientists with research questions rather than as partisans armed with ready-made talking points. Conducting a field experiment demonstrates that scholars do not begin with answers. Instead, we use transparent and systematic methods to examine evidence that can support verifiable claims. 

My other hope was that the hands-on process of designing and implementing a field experiment would encourage maximum buy-in and classroom participation.

The Experiment

My inspiration for this class project was to conduct a streamlined version of a study that Ben Epstein and I co-authored analyzing campaign emails of the 2016 presidential candidates.

Our semester began in late August when the general election campaign was already underway. For this reason, I instructed students to set up their experiment immediately. Data they collected spanned from late August until Election Day on November 5. (No human subjects means no IRB approval needed).

Step 1. The Scientific Method

I assigned Chapter 1 from Alan Gerber and Donald Green’s Field Experiments. It explained research design and why experiments are useful for examining the social world.

Step 2. Email or Text

Students chose to study either campaign emails or text messages. They selected two U.S. House or Senate races from the Cook Political Report ratings that measure competitiveness.

Each student then generated at least one research question and hypothesis. My instructions provided an example of each:

Example Q1: Does political competition shape the frequency and tone of campaign communication?

Example H1: Campaigns in “tossup” races will send emails with greater frequency and more urgent messaging than those in less competitive districts.

Step 3. Set Up

Students signed up on campaign websites to receive either email or text campaign messages from four candidates (two Democrats and two Republicans) across two races.

I required new email or texting accounts with inbox zero for each candidate. Each account received messages from only one candidate. I listed free email and texting services to easily create these accounts.

Step 4. Data Collection

Campaign emails or texts received were the data collected. I recommended that students periodically sign in to each campaign account to ensure there were no problems and that each account was isolated to receiving emails from one candidate (including checking the spam folder).

Step 5. Explain Your Research

Explaining research clearly and concisely is an essential part of social science.

I required students to have one conversation with someone about their field experiment. They had to explain how the experiment was set up and what they hoped to find. Was explaining the research hard? Did it lead them to rethink the project?

To ensure this step occurred, students submitted a form that summarized their conversation, including a signature from the person they spoke to.

Step 6. Data Analysis

After the election was over, students began to organize and analyze data. Students came up with measurements for their original research questions.

I offered suggestions about how to proceed. Depending on a student’s research question, I recommended measuring quantity (how many messages were sent), sender of origin (a party, PAC, or a celebrity) or message timing (by major events or deadlines). Most common was categorizing emails by message content: issue position, fundraising, voter mobilization, or informational.

Some undergraduates do not have confidence with higher-level math. I encouraged them to use basic math skills like adding up totals, and calculating averages and percentages. Students organized their data into tables, charts, and figures.

(Above: anonymized example of one student’s presented data that coded messages by content.)

Step 7. Research Paper   

Following the structure of a research article, students wrote a brief paper that included an introductory section, research design and methods, data collection, findings, and a conclusion.

Step 8. Presentation

Lastly, they delivered a five-minute talk to the class with slides that explained the topline findings from their experiments. Presenters answered conference-style questions for an additional two minutes.

How It Went

Students discovered that running their own experiment was fun and exciting. Several people who rarely spoke in class or had weak performance on exams excelled on this project.

As anticipated, studying the 2024 election through an experimental framework focused class attention on how to measure, interpret and present data rather than on taking sides in partisan polemics.   

Overall, this experiment was far more successful than I had expected. Students felt like real political scientists because they were.

Additional Considerations

There were hiccups, of course.

One issue that arose was that most students chose to study email rather than text communications; however, campaigns now use email to reach supporters less frequently than by text or social media.

A small number of students received no campaign emails whatsoever to one or two accounts, reducing the number of data points. Null results are far from a disaster in field experiments. Still, it did make students nervous. Some students worried they had made an error in the set-up stage (they didn’t) or that they might receive a lower grade. I assured them fewer data points would not impact their grade.  

Students also commented that email was too outdated. They preferred following campaign social media accounts. It would be possible to substitute social media accounts for email in research design. If you bring social media into a classroom studying elections, however, it also carries an array of teaching minefields and ethical quandaries.