Tag: Research Excellence Framework

Research Excellence Framework

I teach a session on the politics of research funding, offered now as part of our workshop and seminar series for postgraduate research students in the School of Healthcare Sciences at Cardiff University. One of the things I talk about is the Research Excellence Framework (REF), and how the outcomes from the REF are used to inform the annual distribution of quality-related (QR) funding to universities as part of the ‘dual support’ approach, along with providing evidence of a return on public investment and generating information for the purposes of benchmarking and comparison.

The REF moves in long cycles. The first took place in 2014, replacing the Research Assessment Exercise which had previously run every five or so years from the middle of the 1980s to 2008. For the 2021 Research Excellence Framework (REF2021) universities made submissions to up to 34 units of assessment, each covering a subject area and each operating underneath one of four main panels. Most research conducted by nursing academics in the REF2021 census period will have been submitted to the Allied Health Professions, Dentistry, Nursing and Pharmacy unit of assessment (UOA). In each UOA, expert panels reviewed the quality of three separate components within each submitting university’s return: outputs, impact and environment. An output might be a single journal article, or perhaps a book or monograph, assessed by members of the relevant expert panel for its ‘originality, significance and rigour’. The assessment of impact involved the expert review of university-submitted case studies exemplifying the ‘reach and significance’ of research beyond academia. Environment involved review of how far each university was enabling the ‘vitality and sustainability’ of excellent research in the subject area.

Suffice to say, the REF is a big deal for universities, subject areas and researchers, with very considerable resources being devoted by institutions to secure as favourable a set of outcomes as possible. As a process it has both its supporters and its detractors, with a summary of controversies surrounding the REF appearing on this Wikpedia page. Debates aside, results from REF2021 have now been published, with the data searchable both by university and by unit of assessment. Main panel reports have also appeared, with more data (including the detail of each submission to each UOA) expected in due course. Both the Times Higher and the Wonkhe teams have produced independent analyses of what the results mean, with the Council of Deans of Health publishing a response highlighting how ‘research undertaken by our members [as submitted to the Allied Health Professions, Dentistry, Nursing and Pharmacy UOA] tackles real life health challenges facing patients, health services and local communities as well as addressing wider societal challenges and global health inequalities.’

Extracted directly from the Overview report by Main Panel A and Sub-panels 1 to 6, here is a summary of the average assessed quality profile of research submitted to the Allied Health Professions, Dentistry, Nursing and Pharmacy UOA:

Reproduced from the Overview report by Main Panel A and Sub-panels 1 to 6

Here, 4* refers to ‘world-leading’, 3* to ‘internationally excellent’, 2* to ‘internationally recognised’ and 1* to ‘nationally recognised’. Overall, then, this table gives lots to celebrate, pointing to allied health, nursing and related research being done to a very high quality, and making a significant difference beyond academia. This much is acknowledged explicitly in the UOA-specific overview report, but so too is a note on the importance of continued investment in research capacity-building and in supporting early careers. I find it hard to disagree with that sentiment. Whilst REF outcomes come round only once every six or seven years, the work of enabling and growing research capability is a constant.

Conferences and meetings catch-up

October has been a month of external events. These began with the inaugural All Wales Mental Health Nursing and the #FutureMHN conferences, in Cardiff and Birmingham respectively. The former was organised through the All Wales Senior Nurse Advisory Group for Mental Health with support from Public Health Wales. The latter was the third such event of its kind, ably organised by a student-committee led. I’ve since been to Derry for the Autumn term meeting of Mental Health Nurse Academics UK and for Ulster University’s 14th annual mental health conference, the latter organised under the theme of Quality and Compassion: Challenges and Opportunities for Mental Health. 

At the MHNAUK meeting Professor Hugh McKenna, who chaired the #REF2014 Allied Health Professions, Dentistry, Nursing and Pharmacy Panel, gave a talk on #REF2021. We livestreamed this, and have embedded a link in the MHNAUK website. Scrolling to the bottom of this page takes you there.

In with the new

Following a discussion involving the event’s steering committee and Mental Health Nurse Academics UK (MHNAUK), what was the International Network for Psychiatric Nursing Research Conference has now become the International Mental Health Nursing Research Conference. Next year we meet in Cardiff on September 14th and 15th, and early information can be found here. This site will be updated as further plans are made, so it’s worth checking in from time to time. We’ve also updated our conference twitter feed; clicking takes you there. Our hashtag, which is already in use, is #MHNR2017.

For a taster of the conference, here are links to four of last year’s keynote presentations: Steven Pryjmachuk, Elaine Hanzak, Luciana Berger MP and Bryn Lloyd-Evans:

Meanwhile, I’m pleased to have been elected as Vice Chair of MHNAUK. I’ll be working with the Chair, Steven Pryjmachuk, throughout 2017 and 2018 before becoming Chair at the start of 2019 for a further two years. I’m grateful for the opportunity colleagues have given me, and will do my best to lead and represent the group and the wider field. I anticipate having plenty to do, noting changes ahead in the education of nurses, the funding of students and the run-in to the next research excellence framework. More on all these to follow, I’m sure: with time today to draw attention to the decision by the Welsh Government to continue bursary payments in 2017-18 for eligible students of nursing, midwifery and the allied health professionals. As the press release announcing this makes clear, individuals taking this offer up will need to commit ‘in advance to taking up the opportunity to work in Wales, post qualification, for a period of two years’. Entirely unclear are funding arrangements for the period from 2018-19 onwards.

Higher education Green Paper

The much-trailed higher education Green Paper appeared last week. For the full document, the place to go is here and for a Times Higher summary the link is here. Immediately worth bearing in mind is that higher education in the UK is a matter for devolved government, meaning that most of what the Green Paper says relates to English universities. I say ‘most’ because, as is noted towards the beginning of the document, the Research Councils (like the MRC and the ESRC) have UK-wide remits, whilst the REF and the various editions of the RAE preceding it were carried out on a four-country basis. It would also be naive in the extreme to suggest that universities and policymakers here in Wales can ignore the England-only bits of what the Green Paper has to say; for a nice piece on the Green Paper and devolution, follow this link. And, for an insight into work ongoing in Wales on matters higher education-related, here’s a link to a current review of funding arrangements.

Fulfilling our potential: teaching excellence, social mobility and student choice, to give the Green Paper its full title, proposes plenty of change. It also leaves much of the detail unfilled, and (in at least one analysis) contributes to an emerging higher education policy framework which is both vague and contradictory. One idea is for the Higher Education Funding Council for England and the Office for Fair Access to combine functions within a new Office for Students. Plans for a Teaching Excellence Framework are outlined, along with variable rates of tuition fees. Ahead of the publication of the document there was some talk that REF2014 might have been the last of its kind. What the Green Paper actually says (briefly) is that dual support for research should remain, and that some version of the research excellence framework should continue and be used as the basis for the allocation of government block funding. The next REF, it is suggested, will take place by 2021. But there is obviously more to follow in this context, with sections in the document referring to the administrative burden and cost of the REF and the possible use of metrics to ‘refresh’ quality assessments in between full cycles of peer review. There is also the small matter of having to determine, if the Higher Education Funding Council for England disappears, which body should in the future assume the task of allocating quality-related funding to English universities.

Meanwhile, the sole nurse to get a mention in the Green Paper and its surrounding commentary is Sir Paul Nurse, Nobel Laureate, President of the Royal Society and chair of the review into the UK’s research councils. Nurses, of the type of which I am one, have to look elsewhere for the specifics on possible future arrangements for the organisation and funding of health care professional education and for debates in this area.

REF results out, COCAPP in

The publication of results from the Research Excellence Framework 2014 (#REF2014) has made this a big week for universities. REF is important for lots of reasons. First, it is assumed that recurring, quality-related (QR), research funding from the UK’s four higher education funding councils will be weighted to the REF results (I say ‘assumed’ here in the light of reports, today, on possible changes to the dual support system: see below for more). Second, universities and the departments within them are ranked based on REF outcomes, making the exercise a crucial one for relative reputations. Third, for governments the REF (like the various research assessment exercises (RAEs) before it) is a way of showcasing the value of research investment and the wider benefits this brings.

Universities will have followed different strategies in managing their REF submissions. In Cardiff a selective return produced a result comfortably better than had been aimed for with the University now ranked in the top five based on research quality. Check out this short video for an overview:

As I hinted above there are, already, questions being asked of how the REF results will (or will not?) be converted into future funding. A report in today’s Observer suggests that changes may be afoot to the dual support (QR and programme/project-related) system. Here’s an early morning tweet from Phil Baty at the Times Higher Education:

So that’s an evolving story which deserves to be closely watched.

Meanwhile…

…moving from research in the round to research projects specifically, this has also been the week that our draft final report from COCAPP has been submitted for peer review to the NIHR Health Services and Delivery Research Programme:

COCAPP has been an investigation into care planning and care coordination in mental health services, and has already been partnered by COCAPP-A. This related study is asking questions in the hospital setting similar to those asked by COCAPP in the community. The coming year sees COCAPP-A getting into full swing, with qualitative and quantitative data being generated across multiple NHS sites in England and Wales.

Opening access

At June 2014’s meeting of the School of Healthcare Sciences’ Research and Innovation Committee there is an agenda item on the UK higher education funding councils’ new policy for open access in the post-2014 Research Excellence Framework. This document contains important information for UK academics with aspirations for future REF return. Here’s a snip from the opening pages, addressing the new requirements for journal articles:

[…] to be eligible for submission to the post-2014 REF, authors’ final peer-reviewed manuscripts must have been deposited in an institutional or subject repository on acceptance for publication. Deposited material should be discoverable, and free to read and download, for anyone with an internet connection. […] The policy applies to research outputs accepted for publication after 1 April 2016, but we would strongly urge institutions to implement it now.

Elsewhere the policy states that publishers’ embargo periods before final versions of papers are deposited can be respected, but with limits. For articles included in submissions to REF Main Panel A (and that includes Nursing), the maximum time period before REF-eligible papers must be made freely available in either green or gold open access format will be 12 months.

There are circumstances in which future REF panels can make exceptions to these rules, but (by the looks of things) not many. In any case, plenty of publishers and individual journals already use copyright transfer agreements which allow authors to comply. It is precisely because of these agreements that I have been able to freely deposit full-text, post-peer review, green open access versions of many of my publications in Cardiff University’s Orca repository and to include links to them on this blog.

But not all journals are currently using copyright transfer agreements which adhere to these new rules. Here’s an example. Nicola Evans’ and my recent paper on critical junctures appears in Social Theory & Health, which is published by Palgrave Macmillan. The journal’s current copyright transfer arrangement (which we signed) allows authors to deposit a post-peer review version of their accepted articles to a repository, but only after an embargo period of 18 months. As things stand, this fails to meet the post-2016 requirement for articles eligible for return to Main Panel A in the next REF. As Palgrave will surely want REF-aspirant nurses (and others) to continue submitting papers to its journals this embargo period will have to be reduced by at least six months.

And then there are the journals published by Wiley Blackwell, including the Journal of Psychiatric and Mental Health Nursing. I’ve published lots in this over the years, most recently on work and roles. The relevant copyright transfer agreement can be viewed here, in which I see that authors retain the right to:

[…] self-archive the peer-reviewed (but not final) version of the Contribution on the Contributor’s personal website, in the Contributor’s company/institutional repository or archive, and in certain not for profit subject-based repositories such as PubMed Central as listed at the following website: http://olabout.wiley.com/WileyCDA/Section/id-820227.html, subject to an embargo period of 12 months for scientific, technical and medical (STM) journals and 24 months for social science and humanities (SSH) journals following publication of the final Contribution.

So peer reviewed, but not ‘final’, versions of accepted papers can be deposited? But it’s the ‘final’ versions which the funding councils specify must be made available. But might ‘final’ mean different things to different people? If Wiley uses ‘final’ to refer to the formatted, pdf, versions of papers identical in every way to the versions appearing in its journals, then authors depositing their word-for-word, post-peer review, author-accepted green open access versions will be compliant. But if ‘final’ for Wiley refers to the accepted, word-for-word (but not necessarily value-added) versions of papers then the inability of authors to make green open access deposits becomes a problem from 2016. So perhaps this all needs some clarification.

Returning to the REF

Photo by Antony Theobald (ant.photos) Creative Commons 2.0 (CC BY-NC-ND 2.0) licence

This is the month that universities in the UK make their submissions to the Research Excellence Framework (REF) 2014. The REF is a big deal, as I’ve written about before. It is also continuing to attract plenty of commentary, much of it critical. For some time Dorothy Bishop, Professor of Developmental Neuropsychology at Oxford University, has been using her personal blog to critique exercises in research rating. Her objections include their poor cost-effectiveness and the dangers of using journal impact factors as a proxy for the quality of individual papers. In his blog Peter Coles, Professor of Theoretical Astrophysics at Sussex University, has attacked the REF for becoming self-serving. Quoting from a Times Higher Education (THE) story he also writes of the practice in some universities of research-active academics not selected for REF return being shifted onto teaching-only contracts. This week, Professor Peter Scott from the Institute of Education writes in The Guardian that research assessment is now ‘out of control’ whilst the THE has recently reported on the case of Lancaster University historian Professor Derek Sayer who has appealed against the decision to include him in the REF on the grounds that the procedures used to exclude some of his colleagues have been discriminatory.

And so it goes on. In the REF proper, outputs (typically articles in journals) will be graded by experts as ‘world leading’ (4*), ‘internationally excellent’ (3*), ‘internationally recognised’ (2*), ‘nationally recognised’ (1*) or as either ‘sub-national’ or ‘not research’. These gradings will be made using the criteria of originality, significance and rigour. Universities get to select which of their staff will be included in their returns, drawing on their preparatory assessments of the quality of eligible outputs and underpinned by strategic ambitions of where they want to be in the HE firmament once the official REF results are published and institutions ranked. My guess is that, for most researchers and their employers, the most important distinction needing to be made will have been between outputs which are internationally excellent (3*) and outputs which are ‘only’ internationally recognised (2*). For reasons of reputation and likely future funding an article assessed as being at least the former is much more likely to be included in a REF submission than one which is not.

Quality assessments informing imminent REF returns will have been made by busy people with varying degrees of expertise in the (sub)areas in which the papers they have been reading lie. I’m going to speculate that there will be many hundreds (thousands?) of academics with outputs which will have attracted inconsistent scores from internal and external reviewers. Who knows, perhaps there are even some with individual outputs assessed by different people as simultaneously being ‘world leading’ and ‘unclassified’. Many will certainly have papers differentially judged as being 2* or 3*, leaving all sorts of tricky decisions to be made on submission or non-submission with all manner of possible consequences for both individuals and universities.

Back in the world of actually doing research, as opposed to the world of assessing research outputs and fretting over returns to assessment exercises, I am pleased to say COCAPP is now receiving questionnaires from service users and RISC is deep into phase 2. If you head over to this NISCHR page you’ll also find news of the Plan4Recovery project, led by Michael Coffey. This is a collaboration involving Hafal, and I’m very pleased to be a co-applicant along with Sherrill Evans and Alan Meudell. Plan4Recovery is advertising for a research officer, and is about to have its first advisory group meetings. Exciting times.

Research and mental health nursing

Over on the Mental Health Nurse Academics UK blog, the group’s chair Dr Michael Coffey writes:

The Research Excellence Framework (REF) looms large for many of us. This is part of the regular round of judging of peer-reviewed research publications on which the UK government bases its decisions on distribution of institutional research funding. Decisions within Higher Education Institutions are being made around now on who is considered returnable and then whether it is strategically advantageous to submit these researchers in the exercise. For individual research careers these decisions weigh heavy. For the profession of mental health nursing there may be wider implications. Previous research assessment exercises have seen more and more evidence of mental health nurses being returned. This has undoubtedly led to an improved profile within individual universities and recognition of the contribution of research to improving the experience of people using mental health services.  There remain significant opportunities for mental health nurse researchers to contribute and bring to bear a professional view on what needs researching and how this should be conducted. We will have to wait until 2014 for an idea of what the landscape will look like in relation to mental health nursing. One thing for sure is that we need a highly engaged mental health nurse academic workforce to continue to provide high quality education and research. By doing this we can make a meaningful contribution to the development of mental health care both here in the UK and beyond.

Nursing certainly did do well in the UK’s last Research Assessment Exercise, the precursor to the REF to which Michael refers in his post. Results for all higher education institutions submitting to Unit of Assessment (UoA) 11 (Nursing and Midwifery) in RAE 2008 can be found here. In his subject overview report for UoA11, Professor Hugh McKenna of Ulster University ended with this:

In conclusion, the sub-panel members were very impressed with most of the submissions they reviewed and with the pervasive pattern of world-leading and internationally excellent research. There are many models of good practice from which developing research groupings can learn in terms of research activities, outputs, environment and esteem. It is clear that investment by Governments, funding bodies and universities has increased research capacity and developed research leaders capable of undertaking nursing and midwifery research that is internationally excellent and world leading. These funding streams need to be sustained and enhanced if the upward trajectory and momentum are to continue and if the quality differentials between the strongest and weakest departments are to be addressed.

And, when the results from the last RAE were published in December 2008, The Guardian ran an article titled Nursing research takes its place on world stage. Here it said:

Nursing, for many years medicine’s poor relation, has come of age in the 2008 research assessment exercise (RAE). Academics in the field can justifiably claim to be world-leading in terms of research. Nursing and midwifery was among the subjects with the most highly rated research in the results published today.

Heady stuff indeed, and testimony to years of hard work, strategy, and capacity-building investment. All this does, though, seem an awful long time ago. As Michael reminds us in his MHNAUK post, attention has long since turned to preparations and prospects for REF 2014.

I have written about nursing and the Research Excellence Framework 2014 on this site before, drawing attention to the workload facing members of UoA A3 (Allied Health Professions, Dentistry, Nursing and Pharmacy) and to the challenges of demonstrating and assessing ‘impact’. So, as we hurtle towards the deadline for REF submission, in what shape does UK mental health nursing research find itself?

Readers of this blog will know that this is a question that MHNAUK is also asking, and is seeking an answer to in organised fashion. Professor Len Bowers led a discussion on this at the MHNAUK meeting held in Cardiff last March. When the group reconvened in Liverpool in June, Dr Fiona Nolan asked members for items to include in her planned survey of research activity and capacity.

Whilst we await findings from Fiona’s project in the first instance, my personal view is that there is much to celebrate in mental health nursing research but also room for development. A small number of universities are home to strong and established research groups. Leaders of these have built national and international collaborations across disciplinary and institutional boundaries. They have laboured to secure funding in open competition and to complete and publish studies with real implications for policy, services, education and practice. This is excellent progress, and I think we now need more of this type of activity across more universities. This means people (and I include myself here) extending their ambition, and perhaps being a little bolder. As an example, early next year the European Union’s Horizon 2020 research and innovation programme invites a first round of applications for funding. How excellent it would be for mental health nurses in the UK to be leading, and collaborating on, high-quality bids submitted there.

My more general reading of the field is that, in many universities, mental health nurse researchers are thinly spread. I’ll bet that in most of the sixty-plus universities represented at MHNAUK the number of people predominantly involved in teaching far outstrips the number predominantly engaged in research. Teaching is important – really important – but the lone researcher in a team of teachers is in a tough place indeed. As I cast an eye around the departments I am most familiar with I also wonder where the younger mental health nurse researchers are. How many mental health nurses in their 20s are studying for PhDs? If the answer is ‘not many’, then what should we collectively be doing to make research a viable, and attractive, career proposition for nurses at the start of their careers? How might we nurture a future generation of mental health nursing clinical academics?

Anyway: all is speculation until we have some evidence. The MHNAUK survey, I anticipate, will paint a more rounded and complete picture of the true state of research activity and capacity than will the Research Excellence Framework. The REF, being what it is, is subject to all sorts of inter- and intra-institutional politicking and will produce only a partial view of what’s really going on.

New year…

Cardiff University Colleges and SchoolsHappy new year. 2013 promises plenty. I’m committed to two externally funded research projects, collaborating with outstanding folk located both in, and out, of Cardiff University. In the fullness of time I’ll perhaps blog about these studies when there’s more to say. I’ll be supervising people working on their doctorates, and as always will be teaching and assessing across the range of academic levels. I’ll be working up grant applications (there’s one in the pot at the moment), writing papers (including the one I’ve mentioned before), and contributing to various types of ethics and scientific review processes. I also have a number of external examining roles to fulfil, at doctoral and pre-registration undergraduate level.

In the year ahead I suspect there will be some interesting organisational changes to adjust to as Cardiff University refashions itself, and as the new College and School structure (which I’ve reproduced to the left of this post, with an added oval to highlight where I work) takes shape. As it happens, the University is making headlines at the moment. Just before the Christmas and New Year break Cardiff’s collaboration with the Open University (and others) to develop ‘MOOCS’ (Massive Open Online Courses) was widely reported. As I understand it, MOOCS are free-to-access courses made available via the web to pretty much anyone with use of a computer and an internet connection. I’m not sure how, if at all, people are able to work towards achieving formal academic awards in this way but I very much like the idea of freely available knowledge. Meanwhile, in this week’s Times Higher Education there’s a report on the new Vice Chancellor’s plans to develop the University’s international presence.

REF 2014In 2013 there’s also the small matter of the Research Excellence Framework (REF). I think the REF (like its predecessor the Research Assessment Exercise, or the RAE) is a flawed process, but it remains a (very) big deal for the UK’s universities. In this cycle, formal submissions will be made at the end of the year. Panel members will then have their work cut out in 2014, reading and assessing the quality of outputs (typically, journal papers), judging the impact of completed research beyond the realms of academia (for example, on policy and practice), and reviewing the institutional environment for research activity. Universities will be ranked on the results, and money will flow (or not). For an ambitious, research-led, Russell Group university like Cardiff this is an exercise of great import. It’s also significant for the professions of nursing and midwifery, which have spent the last decades upping their evidence base. In the last RAE, the outcomes of which were made known at the end of 2008, nursing and midwifery research fared pretty well. Let’s hope this can be sustained.

Outside of work I’ll keep running, hoping to stay injury free. As a meticulous record keeper I track my miles. So far for 2013 it’s 22-and-a-bit, and the aim is to manage 1,000 in total. This I achieved in 2012, and more besides. There’s also an increasingly good chance that this year will see Cardiff City climb out of the Football League Championship. I’m liking this, and it’s something I follow (with season tickets) with one of my boys. And, for those interested in the health and well-being angle of all this, check out the work of Alan Pringle and his colleagues on using football as a means to promote mental health, particularly amongst young men. Alan gave a fantastic talk on this at last year’s Network for Psychiatric Nursing Research conference.

That’ll do for now, I think: enough of the rambling.