The study

The posts I link to above summarize the study, but again for those pressed for time: Horne et al. conducted an experiment with three conditions:

• a control condition, receiving a text about bird feeding;

• a condition that represented ‘traditional interventions’, where scientific evidence was presented that vaccines do not cause autism, called the Autism Correction condition;

• and the new intervention, emphasizing the risks of not getting your children vaccinated, called the Disease Risk condition.

They developed their own measure of attitude towards vaccination, and measured that on the first day of the study. On the second day, those participants that returned were randomized, presented with the intervention of their condition, and completed the attitude measure again. They then conclude that

[t]his study shows that highlighting factual information about the dangers of communicable diseases can positively impact people’s attitudes to vaccination. This method outperformed alternative interventions aimed at undercutting vaccination myths.

How the study is flawed

However, this study does not allow that conclusion - let alone the conclusion that scaring people is the way to get parents to vaccinate their children. The study suffers from a number of serious problems:

The manipulations were not valid

The three conditions vary in a number of ways apart from their main topic (autism vs. risk vs. bird feeding).

First, the word count seriously differs. The Disease Risk condition has 344 words; the Autism Correction condition has 251 words; and the control condition has 193 words. These word counts correlate almost perfectly with the mean change in attitude! I used the brilliant WebPlotDigitizer and entered the image of their anova results on the first page:

Figure 1: Image 1 from the original article

This provided the means for the three conditions: 0.25, 0.07, and 0.05, and used R to correlate these means with the word count for each intervention:

### Enter words and means, compute correlation
words <- c(344, 251, 193);
means <- c(0.25, 0.07, 0.05);
cor(words, means);

[1] 0.9554378

### Compute proportion of explained variance
cor(words, means)^2;

[1] 0.9128614

### Generate plot (note: ggplot2 has to be loaded for this to work!)
ggplot2::qplot(words, means) +
  ggplot2::stat_smooth(method = "lm", se=FALSE) +
  ggplot2::ylim(0, .3) +
  ggplot2::xlim(0, 500);

Figure 2: This image shows the regression line for the mean change predicted by the number of words.

More than 90% of the variation in the attitude change is predicted by the word count. I wish I found such strong associations in my research . . . But on a more serious note, the difference that was observed could simply reflect the well-known Elaboration Likelihood Model principle that if recipients are not motivated or competent to process a message very thoroughly, their attitude change is determined by peripheral factors such as, indeed, word count . . .

[UPDATE: I now discuss this more in depth in this post.]

Second, the Disease Risk condition is the only one that contained images. Regardless of their content, these images could explain the differences between condition.

Third, the Disease Risk condition contained the following fragment:

Measles, mumps, and rubella (MMR) vaccine can prevent these diseases. Most children who get their MMR shots will not get these diseases. Many more children would get them if we stopped vaccinating.

Those sentences may influence participants’ Response Efficacy, a psychological variable reflecting how effective a behavior is believed to be to alleviate a threat (in this case, how well vaccination works). Enhancing participants’ Response Efficacy could explain the results of this study, even more so because the dependent ‘attitude’ measure actually measures Response Efficacy, among other variables.

f <- 5.287;
df1 <- 2;
df2 <- 312;
omegasq <- (f - 1) / (f + (df2 + 1) / (df1))
print(omegasq);

[1] 0.0264978

First figure out what to change, then how to change it

The goal of Horne et al. (2015) is very noble, of course, and I completely agree. It is very important to develop effective behavior change interventions to promote vaccination. However, instead of resorting to threatening communication or any other behavior change method and comparing its effectiveness to other methods, it is necessary to first thoroughly understand the problem at hand (Kok et al., 2014). Without a thorough understanding of why some people refuse to vaccinate their children, attempts to attack this problem are shots in the dark. Reasons for refusing childhood vaccination were recently studied in a Dutch sample (Harmsen et al., 2013; the open access publication is here):

Refusal of vaccination was found to reflect multiple factors including family lifestyle; perceptions about the child’s body and immune system; perceived risks of disease, vaccine efficacy, and side effects; perceived advantages of experiencing the disease; prior negative experience with vaccination; and social environment.

If you want to develop an intervention to promote vaccination, why not start with what we know about why people do or don’t get their children vaccinated? Although the situation in the United States will be different, and therefore, different reasons and considerations will predict behavior in the United States, it is unlikely that the situation is so simple that an intervention targeting only risk perception can be effective.

Instead, it is necessary to first identify what should be changed, in the jargon, which ‘beliefs’ and ‘determinants’ predict behavior. I tried to explain how all this works in a relatively accessible open access paper available here (Peters, 2014; also see Kok, 2014, at http://effectivebehaviorchange.com). There are very, very many other tools available for developing effective behavior change interventions, such as the Intervention Mapping protocol (Bartholomew, Parcel, Kok, Gottlieb, & Fernández, 2011).

However, the study by Horne et al. reflect a dangerous tendency: when talking about behavior change, focusing on the change and not first striving to understand the behavior. When asked “How can we change behavior X?”, “How do we get people to quit smoking?”, or “How do we promote exercise among inner-city youth?”, common responses focus on potential strategies to achieve that goal. The response “Well, first let’s make sure we understand the problem” is rarely heard, except from those who have been trained in systematic behavior change. From the perspective of the media, this is understandable. After all, headlines like “Simple intervention can moderate anti-vaccination beliefs, study finds” work better than “Complex evidence- and theory-based multi-component intervention achieves small increase in vaccination behavior”. From researchers or intervention developers, this is a bit harder to understand. After all, they have access to literature where protocols for intervention development are described (like the Intervention Mapping protocol I just mentioned).

Perhaps a part of the problem is that behavior change science is a part of psychology - and everybody’s a psychologist. When two partners are discussing the behavior of their children, they use their common sense to figure out how they should intervene: and similarly, many people use their common sense to think about how to intervene with the behavior of other people. They’re people, after all - they’re just like us. And if you didn’t study psychology, you don’t know that people are often not rational (but almost always think they are), that people often apply very economical but easily misguided heuristics, and that many processes exist to maintain a positive self-image rather than behave in the most optimal way. There does seem to exist a tendency to discuss other people’s behavior based on the assumption that the products of introspection are informative as to others’ mental landscapes.

This is consistent with the outcomes of a recent qualitative study into why fear appeals remained such a popular weapon of choice for intervention developers, politicians, policymakers, and even some scientists (Peters, Ruiter & Kok, 2014). For example, one scientist who was interviewed said:

I would, if I would read, like, “It gives you lung cancer”, then I would stop to think, like, “Well, I don’t think this is such a nice idea.”

And a politician said:

[…] they showed a brain scan with pictures. So one picture of a child who had not consumed alcohol, and another scan of a child who had consumed alcohol. And showed what it does to your brains. Well, they showed that at the meeting, and then people could ask questions. And then I thought, I think that works, because people then start thinking.

It appears there is a strong (and understandable) tendency to assume that introspection is a useful tool for understanding others. However, psychologists of all people should know better. In addition to the decades of research on behavior change methods (which have shown, for example, that threatening people is usually a bad idea), we also have decades of research on understanding behavior, and it’s time we start combining all of this in our efforts to promote health and in this case, vaccination.

[UPDATE: I now go into the validity of their manipulation more in depth in this post.]

Notes

You can take a look at the measurements and interventions yourself if you want - Horne et al. made their measurements and manipulations publicly available (which is very commendable! I wish they’d also made their data available).

References

Bartholomew, L. K., Parcel, G. S., Kok, G., Gottlieb, N. H., & Fernández, M. E. (2011). Planning health promotion programs: an Intervention Mapping approach (3rd ed.). San Francisco, CA: Jossey-Bass.

Carey, R. N., McDermott, D. T., & Sarma, K. M. (2013). The impact of threat appeals on fear arousal and driver behavior: a meta-analysis of experimental research 1990-2011. PloS One, 8(5), e62821. doi:10.1371/journal.pone.0062821

Fishbein, M., & Ajzen, I. (2010). Predicting and changing behaviour: The Reasoned Action Approach. New York, NY: Taylor & Francis.

Harmsen, I. A., Mollema, L., Ruiter, R. A. C., Paulussen, T. G. W., de Melker, H. E., & Kok, G. (2013). Why parents refuse childhood vaccination: a qualitative study using online focus groups. BMC Public Health, 13(1), 1183.

Horne, Z., Powell, D., Hummel, J. E., & Holyoak, K. J. (2015). Countering antivaccination attitudes. Proceedings of the National Academy of Sciences, 201504019. doi:10.1073/pnas.1504019112

Kok, G. (2014). A practical guide to effective behavior change: How to apply theory- and evidence-based behavior change methods in an intervention. European Health Psychologist, 16(5).

Kok, G., Bartholomew, L. K., Parcel, G. S., Gottlieb, N. H., Fernandez, M. E., & Fernández, M. E. (2014). Alternatives to fear appeals: Theory- and evidence-based health promotion, an Intervention Mapping approach. International Journal of Psychology, 49(2), 98–107. doi:10.1002/ijop.12001

Kok, G., Gottlieb, N., Peters, G.-J. Y., Mullen, P. D., Parcel, G. S., Ruiter, R. A. C., … Bartholomew, L. K. (2015). A taxonomy of behavior change methods: an Intervention Mapping approach. Health Psychology Review.

Peters, G.-J. Y. (2014). A practical guide to effective behavior change: how to identify what to change in the first place. European Health Psychologist, 16(4), 142–155.

Peters, G.-J. Y., Ruiter, R. A. C., & Kok, G. (2013). Threatening communication: a critical re-analysis and a revised meta-analytic test of fear appeal theory. Health Psychology Review, 7(Suppl 1), S8–S31. doi:10.1080/17437199.2012.703527

Peters, G.-J. Y., Ruiter, R. A. C., & Kok, G. (2014). Threatening communication: A qualitative study of fear appeal effectiveness beliefs among intervention developers, policymakers, politicians, scientists and advertising professionals. International Journal of Psychology, 49(2), 71–79. doi:10.1002/ijop.12000

Ruiter, R. A. C., Kessels, L. T. E., Peters, G.-J. Y., & Kok, G. (2014). Sixty years of fear appeal research: Current state of the evidence. International Journal of Psychology, 49(2), 63–70.

Webb, T. L., & Sheeran, P. (2006). Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. Psychological Bulletin, 132(2), 249 –268. doi:10.1037/0033-2909.132.2.249