The research excellence framework (REF) evaluates the quality of the research produced by the higher education institutes (HEIs) in the UK in three areas: quality of research outputs, the impact of this research beyond academia, and the research environment. When the impact element was first introduced in REF2014, the weight (importance) of research outputs was decreased from research assessment exercise (RAE) in 2008 (RAE2008) to REF2014 to accommodate for the weight of impact agenda, with the only exception being panel G (that is, a panel that includes an engineering unit of assessments). Similarly, between the REF conducted in 2014 to the next in 2021, the weight of research outputs will have been further decreased by 5% when the weight of the impact element was increased by 5%. The general trend has been to decrease the importance of research outputs and increase the importance of research impact in the allocation of quality-related (QR) research funding. In our research paper (Pinar & Unlu, 2019), we evaluated how the increased importance of the impact agenda is likely to affect the distribution of research income across universities and subject areas.

submissions that had generated higher external income also tend to achieve relatively better scores in impact assessments

To do this, we first examined the association of some quantitative factors (in other words, submission size and external research income generated by submitted units) with the assessed quality of non‐academic impact in REF2014. First, we found that larger submissions in all panels achieved higher scores in impact element where the effect of submission size is higher in panels B (physical sciences, engineering and mathematics) and C (social sciences) than the ones in panels A (medicine, health and life sciences) and D (arts and humanities). Second, the non-QR elements of the Global Challenges Research Fund and Industrial Strategy Challenges Fund, as well as funding allocated by the UK research councils, also prioritised the non-academic impact in their allocation of funds where we found that the submissions that had generated higher external income also tend to achieve relatively better scores in impact assessments. The non-academic impact will play a significant role on future research income distribution among universities and we expect to see increased research income inequality among HEIs with most of the funding being allocated to a fewer number of universities.

In the second part of the analysis, to examine how units and institutes were affected from the inclusion of impact agenda in the REF, we evaluated the allocation of research funding in the 2017–2018 period if one were to distribute QR funding based on the decisions taken in REF2021, where 25% of QR funding will be distributed in the impact pot, 60% in the output pot and the remaining 15% in the environment pot.

Some of the universities that benefitted from the inclusion of the impact in the REF (that is, generated more QR funding) were University College London (UCL), University of Leeds, University of Manchester, University of Nottingham and University of Bristol, and the ones that were hurt the most were the University of Cambridge, University of Warwick, University of Oxford, University of Birmingham, University of Surrey and University of Exeter (see table S1 of Pinar & Unlu (2019) for the detailed distribution of funds across the institutes).

some subject areas were disproportionately disadvantaged with the inclusion of the impact agenda

Our findings also suggest that some subject areas were disproportionately disadvantaged with the inclusion of the impact agenda. We find that subject areas in panels A, C and D were the main beneficiaries and subject areas in panel B were the main losers. This could be due to many reasons. It is possible that impact on a non-academic community is observed in longer periods in some subject areas such as physics and mathematics (see for example Penfield, Baker, Scoble, & Wykes, 2014). Another explanation would be that the majority of the academic staff members think that links to impact are less straightforward or obvious to evidence in some fields except medical fields (Wilkinson, 2019) and carrying out such an exercise in REF2014 may have disproportionately disadvantaged some fields. Overall, our expectation is to see a lower number of submissions in some subject areas in the next REF exercise, or research active staff members in these units may be returned in another unit of assessments.

Even though the intention of the funding bodies is to increase the societal impact of research by increasing the importance of the impact element, we argue it may have negative consequences on the research carried out in some of the science, technology, engineering and mathematics (STEM) subjects and would also lead to a further increased gap among subjects and institutes.

This blog is based on the article ‘Evaluating the potential effect of the increased importance of the impact component in the Research Excellence Framework of the UK’ by Mehment Pinar and Emre Unlu, published in the British Educational Research Journal.

References

Penfield, T., Baker, M., Scoble, R., & Wykes, M. (2014). Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, 23(1), 21–32.

Pinar, M., & Unlu, E. (2019). Evaluating the potential effect of the increased importance of the impact component in the Research Excellence Framework of the UK, British Educational Research Journal. Advance online publication, https://doi.org/10.1002/berj.3572.

Wilkinson, C. (2019). Evidencing impact: A case study of UK academic perspectives on evidencing research impact, Studies in Higher Education, 44(1), 72–85.