What makes a REF paper REF-able?

About how to improve the outputs written for REF 2021 in the UK

1. What makes a REF paper REF-able? Roger Watson Professor of Nursing Editor-in-Chief, Journal of Advanced Nursing Editor, Nursing Open

2. REF myths Outputs have to be related to one another Impact factors of journals are important Outputs have to be highly cited Order of authorship matters The scoring of my outputs will be published There’s no information available about REF processes If only we could see examples of environment & impact

3. REF 2020 Assessment criteria for outputs unlikely to change Possibly only 4* research will be funded Peer review will be the basis of the assessment

4. 191,150 outputs! 10,358 to sub-panel 3

5. Sub-panel working methods Sub-panels will review their expertise to ensure appropriate coverage Work will be allocated to members/assessors with appropriate expertise Each sub-panel will run calibration exercises for outputs and impacts, guided by the main panels All outputs will be examined in sufficient detail to contribute to the formation of the outputs sub-profiles Each case study will normally be assessed by at least one academic and one user Graduated sub-profiles will be formed for each aspect of submissions REF panels:

6. Research outputs Outputs may include but are not limited to: printed or electronic publications, materials, devices, images, artefacts, products, buildings, confidential or technical reports, patents, performances, exhibits or events All types of research and all forms of research output shall be assessed on a fair and equal basis Panels will assess outputs through a process of expert review. Where stated in the ‘panel criteria’, panels will take account of additional information and/or citation data to inform judgements based on expert review Panels will not use journal impact factors, rankings or lists or the perceived standing of the publisher Outputs:

7. Co-authorship A co-authored output may be listed against one or more individuals that made a substantial research contribution to it It may be listed against any or all such co-authors returned in different submissions ; and a maximum of two such co-authors within the same submission In very specific situations the panels require information to confirm that the author made a substantial research contribution Once this is accepted, panels will assess the quality of the output, not the individual author’s contribution Outputs:

8. Assessment criteria The criteria for assessing the quality of outputs are originality, significance and rigour Each panel provides further explanation of how they will interpret these criteria Panels will assess the quality of outputs, not the contribution of individual researchers to the submission They will examine all outputs in sufficient detail to contribute to the formation of a robust outputs sub-profile that represents all the outputs listed in a submission Outputs:

9. Assessment criteria The criteria for assessing the quality of outputs are originality, significance and rigour* Four star Quality that is world-leading in terms of originality, significance and rigour Three star Quality that is internationally excellent in terms of originality, significance and rigour but which falls short of the highest standards of excellence Two star Quality that is recognised internationally in terms of originality, significance and rigour One star Quality that is recognised nationally in terms of originality, significance and rigour Unclassified Quality that falls below the standard of nationally recognised work. Or work which does not meet the published definition of research for the purposes of this assessment * Each main panel provides descriptive account of the criteria Outputs:

10. How it works Independent peer review of outputs Entered separately on REF system Anomalies discussed and scores agreed Scores are aggregated and individual scores are obliterated from the system No paper trail is maintained Highly reliable system

11. Assessors ~ 300 outputs each as first assessor; plus ~ 300 as second assessor ~ 6 months to complete assessment How long can an assessor spend on an output?

12. Anomalous outputs Books Chapters Double counts Double claims Local reports DON’T!

13. Criteria Originality Significance Rigour Not a ‘tick-box’ exercise Not a peer-review of the article It is a peer assessment of the research

14. Citations and journal rankings Citation data available Journal rankings never enter discussions

15. Writing REF-able papers 1 Remember – it is the research that is being judged The output is only the vehicle However, the vehicle should be in ‘good shape’

16. Writing REF-able papers 2 Use the words ‘originality’, ‘significance’ and ‘rigour’ Use the words ‘international’, ‘global’ Good journals will tend to help you write good papers Use journal facilities such as summaries to convey the main message of the output Don’t use predatory journals

17. Writing REF-able papers 3 Use Equator guidelines to present papers, eg : CONSORT for RCTs PRISMA for Systematic reviews

18. Processes leading up to REF Internal peer review prior to publication Internal REF review External REF review (former sub-panel members) Selection – will depend on university strategy

19. Timing for 2022 Publication date ( ie with doi ) 31 December 2020 Submission to publication can take 12 months Writing can take 12 months (possible including rejections) So: 2021 becomes 2020 (assessment period) 2020 becomes 2019 (final acceptance period) 2019 becomes 2018 (final submission period) 2018 becomes 2017 (final writing period) Open Access – be systematic in using ‘green route’ repositories

20. REF myths Outputs have to be related to one another Impact factors of journals are important Outputs have to be highly cited Order of authorship matters The scoring of my outputs will be published There’s no information available about REF processes If only we could see examples of environment & impact

21. [email protected] 0000-0001-8040-7625 @rwatson1955