Being a fan of scientific research, I often complete surveys. I recently received a survey by email that asked me to choose between fictional medical treatment options — A and B. In each of the survey’s scenarios, figures for the risk of infection, cancer, and surgery associated with each treatment option were provided. A screen capture of one of these scenarios in the survey is below.
The survey was sent to me by physicians at a premier public university in the USA who were acting as the principle investigators for the project, which was sponsored by a national non-profit patient advocacy organization. One assumes that the survey was seen by many sets of highly-educated eyeballs before it was distributed to the public, so I was stunned to see that the survey had a very basic methodological flaw.
The risk for infection and cancer was always stated as “increased chance” while the risk for surgery was defined as “chance.” In the example above, we know that the chance, or absolute risk, of surgery from either A or B is zero. But we do not know the absolute risk — the base rate — of infection or cancer for A or B, because the information is not provided. We are only informed of the increased chance, which is the change in relative risk. The survey ignores the fact that absolute risk must be known prior to making a meaningful decision regarding relative risk in any context. In other words, it’s a situation of base rate neglect.
The inability of highly-educated medical professionals to understand and thus be able to effectively communicate probability-based risk to patients is not unusual. Gerd Gigerenzer, a director at the Max Planck Institute for Human Development, has written about the subject in his book Risk Savvy and in this article in the British Medical Journal. The problem is not limited to physicians either. As Daniel Kahneman has discussed, the human mind, regardless of its intelligence, has difficulty thinking in probabilistic terms.
The badly designed survey led me to thinking about methods, statistics, decision making, psychology, and behavioral economics. I don’t claim to be an expert in any of these topics, nor do I teach courses on them, but I believe they are highly relevant to people’s lives. I’m always on the lookout for resources that might help me demonstrate this relevance to students. In addition to the work of Kahneman and Gigerenzer, I can recommend Charles Wheelan’s Naked Statistics, anything by cognitive psychologist Daniel Willingham, and Adam Smith, Behavioral Economist by Nava Ashraf, Colin F. Camerer and George Loewenstein (Journal of Economic Perspectives 19, 3 Summer 2005).