In a previous blog (published yesterday) I reflected on the danger of researchers going beyond the evidence in presenting policy-relevant findings: putting forward empirical conclusions as cogent when they are supported by weak evidence at best, and/or drawing evaluative conclusions about what is wrong and what should be done. Here I want to discuss a related danger: media reports of research findings that go beyond the evidence, and the role of academics in such reports.

An illustration is provided by a recent study concerned with the effect of school closures on the current Covid-19 pandemic (Viner, 2020). The research took the form of a ‘rapid systematic review’, but since there were few studies looking at the effects of school closure on the spread of Covid-19 itself, the authors had to rely on investigations of closures’ effects on other epidemics, including other forms of Coronavirus such as Sars and Mers. They draw inferences from these studies about the likely effect on Covid-19, taking account of what is known about the latter’s distinctive characteristics. However, they focus on the effect of school closures on their own, separate from other social distancing measures, which is not the situation at present in the UK or in many other countries. This affects not just their claims about the impact of school closure on the spread of the virus but also their assessment of its economic and other costs. In their abstract, the authors’ main practical conclusion is that ‘policy makers need to be aware of the equivocal evidence when considering school closures for Covid-19’ (Viner, 2020, p. 397).

However, in reporting this study, the Guardian’s headline was, ‘School closures have little impact on spread of coronavirus, study finds’ (Weale, 2020; this headline was modified slightly the day after its initial online publication). And in the body of the report it was stated that

‘school closures are likely to have a relatively small impact on the spread of Covid-19 and should be weighed against their profound economic and social consequences, particularly for the most vulnerable children, according to a UK study’.

The authors are quoted as saying that

‘the evidence to support the closure of schools to combat Covid-19 is “very weak”, and statistics from influenza outbreaks suggest school closures “could have relatively small effects on a virus with Covid-19’s high transmissibility and apparent low clinical effect on schoolchildren”’.
(Weale, 2020)

The piece goes on to present comments not just from the lead author, reinforcing these points, but also from two other academics. One of these, Robert Dingwall, is quoted as claiming that ‘the UCL study suggested the public health benefits of school closures were not proportionate to the social and economic costs imposed on children and their families’ (Weale, 2020).

What is at issue here, first of all, is the important difference between there being strong evidence, weak evidence, conflicting evidence, or little evidence either way. The authors argue that there is relatively little relevant evidence, and that what evidence does exist is weak and conflicting. The second issue concerns the relationship between knowledge about the effects of a policy and practical judgments about whether that policy should be pursued or stopped. The Guardian’s headline turned a report showing that there is little and conflicting evidence in support of a policy into a claim that the foundation of the policy is false. Dingwall’s quoted comment builds on this to evaluate the policy negatively.

There is no doubt that the study raises an important issue, and the conclusion drawn in the media may well be true. However, it is misleading to present this study as having provided sufficient evidence to establish its truth. In their article the authors do not make that claim: their discussion is careful and qualified, though they do strive to draw whatever policy-relevant information they can from the very limited evidence available. Academics cannot control the media, but they do provide copy – both directly and indirectly – and there is a need to minimise the danger of distorting research findings. Indeed, we live at a time in which clarity about what the available evidence can and cannot tell us is especially important. Lives and livelihoods may depend on it.

Read the earlier companion piece to this blog, ‘Is educational research evidence-based?’, here. 

References

Viner, R. M., Russell, S. J., Croker, H., Packer, J., Ward, J., Stansfield, C., Mytton, O., Bonnell, C., & Booy, R. (2020). School closure and management practices during coronavirus outbreaks including COVID-19: A rapid systematic review. Lancet Child & Adolescent Health, 4(5), p. 397–404.  https://doi.org/10.1016/S2352-4642(20)30095-X

Weale, S. (2020, April 7). School closures have little impact on spread of coronavirus, study finds. Guardian. Retrievd from https://www.theguardian.com/education/2020/apr/06/school-closures-have-little-impact-on-spread-of-coronavirus-study