Simon Linacre Academia, Research, Scholarly Communication

Not Seeing the Wood for the Trees

There is much to learn from literature regarding scholarly communications, not least how to get messages across the divide to the wider public. Simon Linacre finds much food for thought in chewing over a modern classic.

Like many people, I tend to have two or three books on the go at once. This is due in part to different moods suiting different types of books, but also due to some fallacious idea that I will read them more quickly if I do so concurrently rather than one at a time. It also helps tackle larger books, and there are few larger than my recent fiction read, The Overstory by Richard Powers. A true epic intertwining several different stories that all become rooted to the conservation of redwoods in the Pacific Northwest of the US, it encompasses far more than environmental concerns. Liberty, corporate behavior, relationships and child welfare are all covered, as well as the often-blighted route taken by academic researchers.

The professor in question makes a quite literally groundbreaking discovery about how trees can communicate with each other in a forest through root systems, but as soon as she is lauded she then is put down by fellow academics pouring scorn on her ideas and effectively shunning her from her profession. It is only when her ideas are revived and proven again that she comes into her own, but not as an academic expert – rather as a rabble-rouser, polemicist, and leader of like-minded people.

One of the themes hinted at in this story is the impact debate, and how academic research or any kind of uncovered truth can possibly enable real change in the face of corporate hegemony, state bureaucracy and sheer noise created by those arguing for their individual rights to lead their lives as they see fit. The book offers a little hope among huge piles of despair, but is nevertheless an uplifting read due to the force of will exhibited from the main characters. Similarly, one can see that for those academics who raise their head above the parapet and choose impactful research over more popular or recognised research areas, the path towards individual and societal success can appear a long one.

In one telling passage in the book, one character trying to save a tree by living high in its boughs is almost blown off by a helicopter hovering over her perch. Those behind the actions of the helicopter argue they are saving livelihoods by cutting down the trees; those behind the protest say the same, but those livelihoods stretch far into the future and will be worth more than the short-term approach being adopted by the loggers. The parallels for scholarly communication are also stark, as the limited resources that have backed non-actionable research may soon switch to more actionable outcomes. It just needs all stakeholders to see the wood for the trees and move towards a much more impact-focused trail.

Simon Linacre Academic Publishing, China, Predatory, Research

Beware the known unknowns

Following a recent study showing an alarming lack of knowledge and understanding of predatory journals in China, Simon Linacre looks at the potential impact of the world’s biggest producer of research succumbing to the threat of deceptive publications.

That China has achieved something remarkable in its continued growth in research publications is surely one of the most important developments in modern research and scholarly communications. It passed the US in 2018 and all indications suggest it has increased its lead since then, propelled by huge investment in research by the Chinese government.

Cabells sought to reflect on this success when it published the list of top Chinese-language management journals in December 2020 following a collaboration with AMBA. However, research on that project also highlighted the significant risk for Chinese scholars in publishing in the wrong journals. Until last year, academics tended to be pushed towards – and recognised for – publishing in Impact Factor journals. This policy has, however, now changed, with more of a focus on Chinese-language journals as well as other international titles. The concern then arises, that some scholars may be lured into publishing in predatory journals with the shift in policy.

This thought has been fortified by the publication of the article ‘Chinese PhD Students’ Perceptions of Predatory Journals’ (2021) by Jiayun Wang, Jie Xu and DIanyou Chen in the Journal of Scholarly Publishing. Their study looks at the attitudes of over 300 Chinese doctoral students towards predatory journals, making three key findings:

  1. In STEM subjects, students regularly confused predatory journals with Open Access (OA) journals
  2. In Humanities and Social Science subjects, students tended to only identify predatory journals in the Chinese language, but not in English
  3. While the majority of respondents said they had no intention of submitting to predatory journals (mainly due to the potential harm it could do to their reputation), the few that would do so cited quick publication times and easy acceptance as motivating factors.

While there are limitations to the Wang et al article due to its relatively small sample and restricted scope, it is clear there is at least the potential for widespread use and abuse of the predatory publishing model in China, in parallel to what has been observed to a greater or lesser degree around the rest of the world. In conclusion, the authors state:

“PhD candidates in China generally have insufficient knowledge about predatory journals, and also generally disapprove of publishing in them.” (2021, pp. 102)

This lack of knowledge is referred to time and time again in articles about predatory publishing, of which there is now a small library to choose from. While there is considerable debate on how to define predatory journals, how to identify them and even score them, there is a gap where a better understanding of how to prevent publication in them can be engendered, particularly in the PhD and early career scholar (ECR) communities. Some studies on this aspect of predatory publishing would be very welcome indeed.

Simon Linacre Academic Publishing, Predatory publishing, Research, Scholarly Communication

Book review – Gaming the Metrics: Misconduct and Manipulation in Academic Research

The issues of gaming metrics and predatory publishing undoubtedly go hand-in-hand, outputs from the same system that requires academic researchers the world over to sing for their supper in some form or other. However, the two practices are often treated separately, almost as if there was no link at all, so editors Biagioli and Lippman are to be congratulated in bringing them together under the same roof in the shape of their book Gaming the Metrics: Misconduct and Manipulation in Academic Research (MIT Press, 2020).

The book is a collection of chapters that cover the whole gamut of wrongheaded – or just plain wrong – publication decisions on behalf of authors the word over on where to publish the fruits of their research. This ‘submission decision’ is unenviable, as it inevitably shapes academic careers to a greater or lesser degree. The main reason why authors make poor decisions is laid firmly at the doors of a variety of ‘publish or perish’ systems which seek to quantify the outputs from authors with a view to… well, the reason why outputs are quantified is never really explained. However, the reason why such quantification should be a non-starter is well-argued by Michael Power in Chapter 3, as well as Barbara M. Kehm (Ch. 6) in terms of the ever-popular university rankings. Even peer review comes under attack from Paul Wouters (Ch. 4), but as with the other areas any solutions are either absent, or in the case of Wouters proffered with minimal detail or real-world context.

Once into the book, any author would quickly realize that their decision to publish is fraught with difficulty with worrying about predatory publishers lurking on the internet to entice their articles and APCs from them. As such, any would be author would be well advised to heed the call ‘Caveat scriptor’ and read this book in advance of sending off their manuscript to any journals.

That said, there is also a case for advising ‘caveat lector’ before would-be authors read the book, as there are other areas where additional context would greatly help in addressing the problems of gaming metrics and academic misconduct. When it comes to predatory journals, there is a good deal of useful information included in several of the later chapters, especially the case studies in Chapters 7 and 15 which detail a suspiciously prolific Czech author and sting operation, respectively.

Indeed, these cases provide the context that is perhaps the single biggest failing of the book, which through its narrow academic lens doesn’t quite capture the wider picture of why gaming metrics and the scholarly communications system as a whole is ethically wrong, both for those who perpetrate it and arguably the architects of the systems. As with many academic texts that seek to tackle societal problems, the unwillingness to get dirt under the fingernails in the pursuit of understanding what’s really going on simply distances the reader from the problem at hand.

As a result, after reading Gaming the Metrics, one is like to simply shrug one’s shoulders in apathy about the plight of authors and their institutions, whereas a great deal more impact might have been achieved if the approach had been less academic and included more case studies and insights into the negative impact resulting from predatory publishing practices. After all, the problem with gaming the system is that, for those who suffer, it is anything but a game.

Gaming the Metrics: Misconduct and Manipulation in Academic Research, edited by Mario Biagioli and Alexandra Lippman (published Feb. 21 2020, MIT Press USA) ISBN: 978-0262537933.

Mike Bisaccio Academic Publishing, Cabells News, Higher Education, Research, Scholarly Communication

Cabells and scite partner to bring Smart Citations to Journalytics

Cabells, a provider of key intelligence on academic journals for research professionals, and scite, a platform for discovering and evaluating scientific articles, are excited to announce the addition of scite’s Smart Citations to Cabells Journalytics publication summaries.

Journalytics summary card with scite Smart Citations data

Journalytics is a curated database of over 11,000 verified academic journals spanning 18 disciplines, developed to help researchers and institutions optimize decision-making around the publication of research. Journalytics summaries provide publication and submission information and citation-backed data and analytics for comprehensive evaluations.

scite’s Smart Citations allow researchers to see how articles have been cited by providing the context of the citation and a classification describing whether it provides supporting or disputing evidence for the cited claim.

The inclusion of Smart Citations adds a layer of perspective to Journalytics metrics and gives users a deeper understanding of journal activity by transforming citations from a mere number into contextual data.

Lacey Earle, executive director of Cabells, says, “Cabells is thrilled to partner with scite in order to help researchers evaluate scientific articles through an innovative, comparative-based metric system that encourages rigorous and in-depth research.”

Josh Nicholson, co-founder and CEO of scite says of the partnership, “We’re excited to be working with Cabells to embed our Smart Citations into their Journalytics summaries. Smart Citations help you assess the quantity of citations a journal has received as well as the quality of these citations, with a focus on identifying supporting and disputing citations in the literature.”

about cabells

Cabells generates actionable intelligence on academic journals for research professionals.  On the Journalytics platform, an independent, curated database of more than 11,000 verified scholarly journals, researchers draw from the intersection of expertise, data, and analytics to make confident decisions to better administer research. In Predatory Reports, Cabells has undertaken the most comprehensive and detailed campaign against predatory journals, currently reporting on deceptive behaviors of over 14,000 publications. By combining its efforts with those of researchers, academic publishers, industry organizations, and other service providers, Cabells works to create a safe, transparent and equitable publishing ecosystem that can nurture generations of knowledge and innovation. For more information please visit Cabells or follow us on Twitter, LinkedIn and Facebook.

about scite

scite is a Brooklyn-based startup that helps researchers better discover and evaluate scientific articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or disputing evidence. scite is used by researchers from dozens of countries and is funded in part by the National Science Foundation and the National Institute of Drug Abuse of the National Institutes of Health. For more information, please visit scite, follow us on Twitter, LinkedIn, and Facebook, and download our Chrome or Firefox plugin. For careers, please see our jobs page.

Simon Linacre Academic journals, Academic Publishing, Research, Scholarly Communication

Cabells and AMBA launch list of most impactful Chinese language management journals

In his last blog post in what has been a tumultuous year, Simon Linacre looks forward to a more enlightened 2021 and a new era of open collaboration and information sharing in scholarly communications and higher education.

In a year with so many monumental events, it is perhaps pointless to try and review what has happened. Everyone has lived every moment with such intensity – whether it be through 24-hour news coverage, non-stop social media or simply living life under lockdown – that it seems simply too exhausting to live through it all again. So, let’s fast forward to 2021 instead.

While some of the major concerns from 2020 will no doubt remain well into the New Year, they will also fade away gradually and be replaced by new things that will demand our attention. Difficult as it may seem now, neither Trump, Brexit (for the Brits) nor COVID will have quite the hold on the news agenda as they did, and that means there is an opportunity at least for some more positive news to start to dominate the headlines.

One activity that may succeed in this respect is the open science agenda. With a new budget agreed upon by the European Research Council and a new administration in Washington DC, together with an increasing focus more generally on open science and collaboration, it is to be hoped that there will be enough funding in place to support it. If the recent successes behind the COVID-19 vaccines show anything it is surely that focused, fast, mission-driven research can produce life-changing impacts for a huge number of people. As others have queried, what might happen if the same approach was adopted and supported for tackling climate change?

In the same vein, information sharing and data analysis should also come further to the fore in 2021. While in some quarters, consolidation and strategic partnerships will bring organisations together, in others the importance of data analysis will only become more essential in enabling evidence-based decision-making and creating competitive advantages.

In this way, the announcement today made by Cabells and the Association of MBAs and Business Graduates Association (AMBA & BGA) brings both these themes together in the shape of a new list of quality Chinese-language journals in business and management. The AMBA-Cabells Journal Report (ACJR) has been curated together by both organisations, using the indexing expertise of Cabells and the knowledge of Chinese journals at AMBA & BGA. Both organisations have been all-too-aware of the Western-centric focus of many indices and journal lists, and believe this is a positive first step towards the broadening out of knowledge and understanding of Chinese-language journals, and non-English journals more broadly.

There have also been policy changes in China during 2020 which have meant less reliance on journals with Impact Factors, and more of a push to incentivise publications in high quality local journals. As such, the ACJR should provide a valuable guide to business school authors in China about some of the top journals available to them. The journals themselves were firstly identified using a number of established Chinese sources, as well as input from esteemed scholars and deans of top business schools. Recommended journals were then checked using Google Scholar to ensure they had published consistently over the last five years and attracted high levels of citations.

The new list is very much intended to be an introduction to Chinese-language journals in business and management, and we would very much welcome input from people on the list so we can develop it further for a second iteration in 2021.

For more information on ACJR, visit https://www.associationofmbas.com/ and https://www.cabells.com/ 

Simon Linacre Academic Publishing, Coronavirus, COVID-19, Predatory publishing, Research

No time for rest

This week The Economist published an article on predatory publishing following collaboration with Cabells. Simon Linacre looks into the findings and points to how a focus on education can avert a disaster for Covid-19 and other important research.

One of the consequences of the all-consuming global interest in the coronavirus pandemic is that it has catapulted science and scientists right onto the front pages and into the public’s range of vision. For the most part, this should be a good thing, as there quite rightly has to be a respect and focus on what the facts say about one of the most widespread viruses there has ever been. However, there have been some moments where science itself has been undermined by some of the rather complex structures that support it. And like it or not, scholarly communication is one of them.

Let’s take the perspective of, say, a mother who is worried about the safety of her kids when they go back to school. Understandably, she starts to look online and in the media for what the science says, as many governments have sought to quell fears people have by saying they are ‘following the science’. But once online, they are faced with a tangled forest of articles, journals, jargon, paywalls and small print, with the media seemingly supporting contradictory statements depending on the newspaper or website you read. For example, this week’s UK newspapers have led on how the reduction of social distancing from 2m to 1m can double the infection rate, or be many times better than having no social distancing – both factually accurate and from the same peer reviewed study in The Lancet.

Another area that has seen a good deal of coverage has been preprints, and how they can speed up the dissemination of science… or have the capability of disseminating false data and findings due to lack of peer review, again depending on where you cast your eye. The concerns represented by media bias, the complexity of information and lack of peer review all combine into one huge problem that could be coming down the line very soon, and that is the prospect of predatory journals publishing erroneous, untested information as research in one of the thousands of predatory journals currently active.

This week Cabells collaborated with The Economist to focus some of these issues, highlighting that:

  • Around a third of journals on both the Cabells Journal Whitelist and Blacklist focus on health, with predatory journals in subjects such as maths and physics number more than legitimate journals
  • Geography plays a significant role, with many more English language predatory journals based in India and Nigeria than reliable ones
  • The average output of a predatory journal is 50 articles a year, although 60% of these will never be cited (compared to 10% for legitimate journals)
  • Despite the like of peer review or any of the usual publishing checks, an estimated 250,000 articles each year are cited in other journals
  • Most common severe behaviors (which are liable to lead to inclusion in the Blacklist) are articles missing from issues or archives, lack of editor or editorial board on the website, and journals claiming misleading metrics or inclusion in well-known indexes.

Understandably, The Economist makes the link between so much fake or unchecked science being published and the current coronavirus threat, concluding: “Cabells’ guidelines will only start to catch dodgy studies on COVID-19 once they appear in predatory journals. But the fact that so many “scholars” use such outlets means that working papers on the disease should face extra-thorough scrutiny.” We have been warned.

Simon Linacre Academic Publishing, Research, Scholarly Communication

Doing your homework…and then some

Researchers have always known the value of doing their homework – they are probably the best there is at leaving no stone unturned. But that has to apply to the work itself. Simon Linacre looks at the importance of ‘researching your research’ and using the right sources and resources.

Depending on whether you are a glass half full or half empty kind of a person, it is either a great time for promoting the value of scientific research, or, science is seeing a crisis in confidence. On the plus side, the value placed on research to lead us out of the COVID-19 pandemic has been substantial, and rarely have scientists been so much to the fore for such an important global issue. On the other hand, there have been uprisings against lockdowns in defiance of science, and numerous cases of fake science related to the Coronavirus. Whether it is COVID-19, Brexit, or global warming, we seem to be in an age of wicked problems and polarising opinions on a global scale.

If we assume that our glass is more full than empty in these contrarian times, and try to maintain a sense of optimism, then we should be celebrating researchers and the contribution they make. But that contribution has to be validated in scientific terms, and its publication validated in such a way that users can trust in what it says. For the first part, there has been a good deal of discussion in academic circles and even in the press about the nature of preprints, and how users have to take care to understand that they may not yet have been peer reviewed, so any conclusions should not yet be taken as read.

For the second part, however, there is a concern that researchers in a hurry to publish their research may run afoul of predatory publishers, or simply publish their articles in the wrong way, in the wrong journal for the wrong reasons. This was highlighted to me when a Cabells customer alerted us to a new website called Academic Accelerator. I will leave people to make their own minds up as to the value of the site, however, a quick test using academic research on accounting (where I managed journals for over a decade, so know the area) showed that:

  • Attempting to use the ‘Journal Writer’ function for an accounting article suggested published examples from STM journals
  • Trying to use the ‘Journal Matcher’ function for an accounting article again only recommended half a dozen STM journals as a suitable destination for my research
  • Accessing data for individuals journals seems to have been crowdsourced by users, and didn’t match the actual data for many journals in the discipline.

The need for researchers to publish as quickly as possible has perhaps never been greater, and the tools and options for them to do so have arguably never been as open. However, with this comes a gap in the market that many operators may choose to exploit, and at this point, the advice for researchers is the same as ever. Always research your research – know what you are publishing and where you are publishing it, and what the impact will be both in scholarly terms and in real-world terms. In an era where working from home is the norm, there is no excuse for researchers not to do their homework on what they publish.

***REMINDER***

If you haven’t already completed our survey, there is still time to provide your opinion. Cabells is undertaking a review of the current branding for ‘The Journal Whitelist’ and ‘The Journal Blacklist’. As part of this process, we’d like to gather feedback from the research community to understand how you view these products, and which of the proposed brand names you prefer.

Our short survey should take no more than ten minutes to complete, and can be taken here.

As thanks for your time, you’ll have the option to enter into a draw to win one of four Amazon gift vouchers worth $25 (or your local equivalent). More information is available in the survey.

Many thanks in advance for your valuable feedback!

Simon Linacre

Simon Linacre Academic Publishing, medical journals, Predatory publishing, Research

Bad medicine

Recent studies have shown that academics can have a hard time identifying some predatory journals, especially if they come from high-income countries or medical faculties. Simon Linacre argues that this is not surprising given they are often the primary target of predatory publishers, but a forthcoming product from Cabells could help them.

A quick search of PubMed for predatory journals will throw up hundreds of results – over the last year I would estimate there are on average one or two papers published each week on the site (and you can sign up for email alerts on this and other scholarly communication issues at the estimable Biomed News site). The papers tend to fall into two categories – editorial or thought pieces on the blight of predatory journals in a given scientific discipline, or original research on the phenomenon. While the former are necessary to raise the profile of the problem among researchers, they do little to advance the understanding of such journals.

The latter, however, can provide illuminating details about how predatory journals have developed, and in so doing offer lessons in how to combat them. Two such articles were published last week in the field of medicine. In the first paper ‘Awareness of predatory publishing’, authors Panjikaran and Mathew surveyed over 100 authors who had published articles in predatory journals. While a majority of authors (58%) were ignorant of such journals, of those who said they recognized them nearly half from high-income countries (HICs) failed a recognition test, while nearly a quarter from low-income to middle-income countries (LMICs) also failed. The result, therefore, was a worrying lack of understanding of predatory journals among authors who had already published in them.

The second article was entitled ‘Faculty knowledge and attitudes regarding predatory open access journals: a needs assessment study’ and authored by Swanberg, Thielen and Bulgarelli. In it, they surveyed both regular and medical faculty members of a university to ascertain if they understood what was meant by predatory publishing. Almost a quarter (23%) said they had not heard of the term previously, but of those that had 87% said there confident of being able to assess journal quality. However, when they were tested by being presented with journals in their own fields, only 60% could, with scores even lower for medical faculty.

Both papers call for greater education and awareness programs to support academics in dealing with predatory journals, and it is here that Cabells can offer some good news. Later this year Cabells intends to launch a new medical journal product that identifies good quality journals in the vast medical field. Alongside our current products covering most other subject disciplines, the new medical product will enable academics, university administrators, librarians, tenure committees and research managers to validate research sources and publication outputs of faculty members. They will also still be backed up, of course, by the Cabells Journal Blacklist which now numbers over 13,200 predatory, deceptive or illegitimate journals. Indeed, in the paper by Swanberg et al the researchers ask faculty members themselves what support they would like to see from their institution, and the number one answer was a “checklist to help assess journal quality.” This is exactly the kind of feedback Cabells have received over the years that has driven us to develop the new product for medical journals, and hopefully, it will help support good publishing decisions in the future alongside our other products.

PS: A kind request – Cabells is undertaking a review of the current branding for ‘The Journal Whitelist’ and ‘The Journal Blacklist’. As part of this process, we’d like to gather feedback from the research community to understand how you view these products, and which of the proposed brand names you prefer.

Our short survey should take no more than ten minutes to complete, and can be taken here.

As thanks for your time, you’ll have the option to enter into a draw to win one of four Amazon gift vouchers worth $25 (or your local equivalent). More information is available in the survey.

Many thanks in advance for your valuable feedback!

Simon Linacre

Simon Linacre Coronavirus, COVID-19, Research, Scholarly Communication

Unintended consequences: how will COVID-19 shape the future of research

What will happen to global research output during lockdowns as a result of the coronavirus?  Simon Linacre looks at how the effect in different countries and disciplines could shape the future of research and scholarly publications.

We all have a cabin fever story now after many countries have entered into varying states of lockdown. Mine is how the little things have lifted what has been quite an oppressive mood – the smell of buns baking in the oven; lying in bed that little bit longer in a morning; noticing the newly born lambs that have suddenly appeared in nearby fields. All of these would be missed during the usual helter-skelter days we experience during the week. But things are very far from usual in these coronavirus-infected days. And any distraction is a welcome one.

On a wider scale, the jury is still very much out as to how researchers are dealing with the situation, let alone how things will be affected in the future. What we do know is that in those developed countries most impacted by the virus, universities have been closed down, students sent home and labs mothballed. In some countries such as Italy there are fears important research work could be lost in the shutdown, while in the US there is concern for the welfare of those people – and animals – who are currently in the middle of clinical trials. Overall, everyone hopes that the specific research into the coronavirus yields some quick results.

On the flip side, however, for those researchers not confined to labs or field research, this period could accelerate their work. For those in social science or humanities freed from the commute, teaching commitments and office politics of daily academic life, the additional time will no doubt be put to good use. More time to set up surveys; more time for reading; more time for writing papers. Increased research output is perhaps inevitable in those areas where academics are not tied to labs or other physical experiments.

These two countervailing factors may cancel each other out, or one may prevail over the other. As such, the scholarly publishing community does not know yet what to expect down the line. In the short term, it has been focused on making related content freely accessible (such as this site from The Lancet: ). However, what we may see is that there is greater pressure to see research in potentially globally important areas to be made open access at the source given how well researchers and networks have seemed to work together so far during the short time the virus has been at large.

Again, unintended consequences could be one of the key legacies of the crisis once the virus has died down. Organizations concerned about how their people can work from home will no doubt have their fears allayed, while the positive environmental impact of less travelling will be difficult to give up. For publishers and scholars, understanding how their research could have an impact when the world is in crisis may change their research aims forever.

Simon Linacre Academic Publishing, Research, Scholarly Communication

The future of research evaluation

Following last week’s guest post from Rick Anderson on the risks of predatory journals, we turn our attention this week to legitimate journals and the wider issue of evaluating scholars based on their publications. With this in mind, Simon Linacre recommends a broad-based approach with the goal of such activities permanently front and center.

This post was meant to be ‘coming to you LIVE from London Book Fair’, but as you may know, this event has been canceled, like so many other conferences and other public gatherings in the wake of the coronavirus outbreak. While it is sad to miss the LBF event, meetings will take place virtually or in other places, and it is to be hoped the organizers can bring it back bigger and better than ever in 2021.

Some events are still going ahead, however, in the UK, and it was my pleasure to attend the LIS-Bibliometrics Conference at the University of Leeds last week to hear the latest thinking on journal metrics and performance management for universities. The day-long event was themed ‘The Future of Research Evaluation’, and it included both longer talks from key people in the industry, and shorter ‘lightning talks’ from those implementing evaluation systems or researching their effectiveness in different ways.

There was a good deal of debate, both on the floor and on Twitter (see #LisBib20 to get a flavor), with perhaps the most interest in speaker Dr. Stephen Hill, who is Director of Research at Research England, and chair of the steering group for the 2021 Research Excellence Framework (REF) in the UK. For those of us wishing to see crumbs from his table in the shape of a steer for the next REF, we were sadly disappointed as he was giving nothing away. However, what he did say was that he saw four current trends shaping the future of research evaluation:

  • Outputs: increasingly they will be diverse, include things like software code, be more open, more collaborative, more granular and potentially interactive rather than ‘finished’
  • Insight: different ways of understanding may come into play, such as indices measuring interdisciplinarity
  • Culture: the context of research and how it is received in different communities could become explored much more
  • AI: artificial intelligence will become a bigger player both in terms of the research itself and how the research is analyzed, e.g. the Unsilo tools or so-called ‘robot reviewers’ that can remove any reviewer bias.

Rather revealingly, Dr. Hill suggested a fifth trend might be the societal impact, and this is despite the fact that such impact has been one of the defining traits of both the current and previous REFs. Perhaps, the full picture has yet to be understood regarding impact, and there is some suspicion that many academics have yet to buy-in to the idea at all. Indeed, one of the takeaways from the day was that there was little input into the discussion from academics at all, and one wonders if they might have contributed to the discussion about the future of research evaluation, as it is their research being evaluated after all.

There was also a degree of distrust among the librarians present towards publishers, and one delegate poll should be of particular interest to them as it showed what those present thought were the future threats and challenges to research evaluation. The top three threats were identified as publishers monopolizing the area, commercial ownership of evaluation data, and vendor lock-in – a result which led to a lively debate around innovation and how solutions could be developed if there was no commercial incentive in place.

It could be argued that while the UK has taken the lead on impact and been busy experimenting with the concept, the rest of the higher education world has been catching up with a number different takes on how to recognize and reward research that has a demonstrable benefit. All this means that we are yet to see the full ‘impact of impact,’ and certainly, we at Cabells are looking carefully at what potential new metrics could aid this transition. Someone at the event said that bibliometrics should be “transparent, robust and reproducible,” and this sounds like a good guiding principle for whatever research is being evaluated.