Education systems around the world increasingly monitor school performance and hold schools to account using student data from high-stakes examinations (OECD, 2008; NFER, 2018). England has been at the forefront of this move to use test-based school accountability systems, with the Department for Education (DfE) adopting a variety of measures for secondary schools over the past 25 years, most recently ‘Progress 8’ (see Leckie & Goldstein, 2017).

Progress 8 measures average pupil progress made between key stage 2 (age 11) and general certificate of secondary education (GCSE) examinations (age 16) and represents a long-called-for improvement over the previous ‘5A*–C’ raw attainment measure (the percentage of children attaining five or more grade C or higher GCSEs on an A*–G grade scale). Nevertheless, concerns have been raised about the validity of the Progress 8 measure for holding schools to account (FFT Education Datalab, 2018; Leckie & Goldstein, 2019; Duff, 2019), with implications for policymakers, schools and parents who use Progress 8 to inform decision-making around schools.

Our research reviewed journal articles, blog posts and policy documents relating to Progress 8, and we complemented this with new analyses of the underlying data to assess the statistical strengths and weaknesses of Progress 8 (Prior et al., 2021). We identified, explored and developed recommendations in response to five areas of concern.

1. Choice of pupil outcome attainment measure.
2. Potential adjustments for prior attainment and background characteristics.
3. Decisions around which schools and pupils are excluded from the measure.
4. Presentation of Progress 8 to users, choice of statistical model and calculation of uncertainty.
5. Issues related to the volatility of school performance over time.

Regarding our first recommendation, we noted that the choice of outcome attainment measure means Progress 8 is heavily weighted (70:30) in favour of the traditional academic subjects of the English Baccalaureate (EBacc). We therefore recommend presenting a less EBacc-focused measure alongside Progress 8 to provide a more holistic picture of school performance, giving schools greater freedom to pursue more varied curricula.

‘We recommend presenting a less EBacc-focused measure alongside Progress 8 to provide a more holistic picture of school performance, giving schools greater freedom to pursue more varied curricula.’

Our second recommendation is to present a pupil background-adjusted version of Progress 8 alongside the current measure (which is only adjusted for prior attainment) to provide a perspective on school performance that better reflects the different contexts and challenges faced by schools.

Our third recommendation is to recognise pupil mobility by making school Progress 8 scores an average of all pupils who attended each school, weighted by their time in each school. This would hold schools accountable for all the pupils they have taught, whereas the current measure only accounts for those on-roll in the examination year.

In terms of our fourth recommendation, our review revealed that schools account for only around a tenth of overall variation in pupil progress. We recommend clearer communication of the limited importance of school Progress 8 scores in general and the importance – or not – of each individual school’s score.

Our fifth recommendation relates to how Progress 8 scores represent very uncertain predictions as to the future performance of schools. We therefore recommend that this limitation is made much clearer to parents choosing schools. We recommend that multiyear averages for Progress 8 are reported alongside the current single-year measure to communicate and address the volatility of Progress 8 scores from year to year.

While we believe that adopting these recommendations would improve Progress 8, it is important that recognition is given to the more general and longstanding concerns with the way school performance data are used to inform school accountability. In particular, the perverse incentives and unintended consequences induced by the high stakes attached to school performance measures: off-rolling pupils, narrowing of the curriculum, teaching to the test, and undue teacher and pupil stress (Foley & Goldstein, 2012; NAHT, 2018).

The disruption to children’s education and examinations caused by Covid-19 has led Progress 8 to be suspended in 2020 and 2021. The DfE will also be prevented from publishing Progress 8 in 2025 and 2026 as the necessary pupil prior attainment measure will not be available (the 2020 and 2021 key stage 2 tests were also cancelled). Now, therefore, is perhaps the time for not just a rethink around Progress 8 but a more radical rethink around the use of school performance data and accountability in England.

This blog is based on the article ‘A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for “Progress 8” raised by COVID-19’ by Lucy Prior, John Jerrim, Dave Thomson and George Leckie, published in the Review of Education on an open-access basis.

References

Duff, L. (2019, January 25). Does Progress 8 need refining to make it fairer? [NFER Blog]. https://www.nfer.ac.uk/news-events/nfer-blogs/does-progress-8-need-refining-to-make-it-fairer/

FFT Education Datalab. (2018, May 14). Value added measures in performance tables: A recap of the main issues for secondary schools [Blog]. https://ffteducationdatalab.org.uk/2018/05/value-added-measures-in-performance-tables-a-recap-of-the-main-issues-for-secondary-schools/

Foley, B. & Goldstein, H. (2012). Measuring success: League tables in the public sector. https://www.thebritishacademy.ac.uk/documents/265/Measuring-success-league-tables-in-public-sector.pdf

Leckie, G., & Goldstein, H. (2017). The evolution of school league tables in England 1992–2016: ‘Contextual value-added’, ‘expected progress’ and ‘progress 8’. British Educational Research Journal, 43(2), 193–212. https://doi.org/10.1002/berj.3264

Leckie, G., & Goldstein, H. (2019). The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England. British Educational Research Journal, 45(3), 518–537. https://doi.org/10.1002/berj.3511

National Association of Head Teachers [NAHT]. (2018). Improving school accountability. https://www.naht.org.uk/Portals/0/PDF’s/Improving%20school%20accountability.pdf?ver=2021-04-27-121950-093/

National Foundation for Educational Research [NFER]. (2018). What impact does accountability have on curriculum, standards and engagement in education? https://www.nfer.ac.uk/media/3032/nfer_accountability_literature_review_2018.pdf

Organisation for Economic Co-operation and Development [OECD]. (2008).

Measuring improvements in learning outcomes: Best practices to assess the value-added of schools. https://doi.org/10.1787/9789264050259-en

Prior, L., Jerrim, J., Thomson, D., & Leckie, G. (2021). A review and evaluation of secondary school accountability in England: Statistical strengths, weaknesses, and challenges for ‘Progress 8’ raised by COVID-19. Review of Education, 9(3), 1–30. https://doi.org/10.1002/rev3.3299