What really counts for rankings?

University and business school rankings have induced hate and ridicule in equal measure since they were first developed, and yet we are told enjoy huge popularity with students. Simon Linacre looks at how the status quo could change thanks in part to some rankings’ own shortcomings.


In a story earlier this month in Times Higher Education (THE), it was reported that the status of a university vis-à-vis sustainability was now the primary consideration for international students, ahead of academic reputation, location, job prospects and even accessibility for Uber Eats deliveries – OK, maybe not the last one. But for those who think students should place such considerations at the top of their lists, this was indeed one of those rare things in higher ed in recent times: a good news story.

But how do students choose such a university? Amazingly, THE produced a ranking just a week later providing students with, you guessed it, a ranking of universities based on their sustainability credentials. Aligned with the UN’s now-ubiquitous Sustainability Development Goals (SDGs), the ranking is now well-established and this year proclaimed the University of Manchester in the UK as the number one university that had the highest impact ranking across all 17 SDGs, although it was somewhat of an outlier for the UK, with four of the top ten universities based in Australia.

Cynics may point out that such rankings have become an essential part of the marketing mix for outfits such as THE, the Financial Times and QS. Indeed the latter has faced allegations this week over possible conflicts of interest between its consulting arm and its rankings with regard to universities in Russia – a charge which QS denies. However, perhaps most concerning is the imbalance that has always existed between the importance placed on rankings by institutions and the transparency and/or relevance of the rankings themselves. A perpetual case of the tail wagging the dog.

Take, for instance, the list of 50 journals used by the Financial Times as the basis for one of its numerous criteria for assessing business schools for its annual rankings. The list is currently under review after not changing since 2016, and then it only added 5 journals from the 45 it used prior to that date, which was itself an upgrade from 40 used in the 2000s. In other words, despite the massive changes seen in business and business education – from Enron to the global financial crisis to globalisation to the COVID pandemic – there has been barely any change in the journals used to assess publications from business schools to determine whether they are high quality.

The FT’s Global Education Editor Andrew Jack was questioned about the relevance of the FT50 and the rankings in general in Davos in 2020, and answered that to change the criteria would endanger the comparability of the rankings. This intransigence by the FT and other actors in higher education and scholarly communications was in part the motivation behind Cabells’ pilot study with the Haub School of Business at St Joseph’s University in the US to create a new rating based on journals’ output intensity in terms of the SDGs. Maintaining the status quo also reinforces paradigms and restricts diversity, marginalizing those in vulnerable and alternative environments.

If students and authors want information on SDGs and sustainability to make their education choices, it is beholden on the industry to try and supply it in as many ways as possible. And not to worry about how well the numbers stack up compared to a world we left behind a long time ago. A world that some agencies seem to want to cling on to despite evident shortcomings.

Opening up the SDGs

While the United Nations Sustainable Development Goals (SDGs) offer a framework for global communities to tackle the world’s biggest challenges, there are still huge barriers to overcome in ensuring research follows the desired path. This week, Simon Linacre reflects on the ‘push’ and ‘pull’ effects in publishing and one organization trying to refine a fragmented infrastructure.

Recently, Cabells has been able to further its commitment to pursuing the UN SDGs by signing up to the SDG Publishers Compact and sharing details of its pilot journal rating system with the Haub School of Business at Saint Joseph’s University that assesses journals in terms of their relevance to the SDGs. Part of the reason Cabells is working with the SDGs – aside from a simple belief that they are a force for good – is that they represent an opportunity to offer reward and recognition for researchers who are using their talents to in some small way make the world a better place.

The scholarly communications industry, like many others, relies on push and pull dynamics to maintain its growth trajectory. The push elements include the availability of citations and other metrics to judge performance, recognition for publishing in certain journals, and various community rewards for well-received research. On the flip side, pull elements include opportunities shared by academic publishers, a facility to record research achievements, and an opportunity to share findings globally. This is how the publishing world turns round.

This dynamic also helps to explain why potentially disruptive developments – such as Open Access or non-peer-reviewed journals and platforms – may fail to gain initial traction, and why they may require additional support in order to become embedded with academics and their mode of operations. Going back to the SDGs, we can see how their emergence could similarly be stymied by the existing power play in scholarly publishing – where are the push and pull factors guiding researchers to focus on SDG-related subjects?

I recently spoke to Stephanie Dawson, CEO at ScienceOpen, which is a discovery platform that seeks to enable academics to enhance their research in an open access environment and offer publishers ‘context building services’ to improve the impact of their outputs. ScienceOpen is very much involved with the UN SDGs, recently creating a number of content containers for SDG-related articles. By offering curative opportunities, post-publication enhancements, and article-level data services, ScienceOpen is most definitely doing its part to support a pull strategy in the industry.

Stephanie says, “We began this project working with the University College London (UCL) Library to showcase their outputs around the UN SDGs. Because we believe there needs to be broad community buy-in, we also wanted to encourage researchers globally to highlight their contributions to the Sustainable Development Goals by adding keywords and author summaries on ScienceOpen, regardless of the journal they published in and demanding publisher engagement for new works.”

And this is what Cabells is also trying to achieve – by offering new metrics that can be used to guide authors to the optimal publishing option (push) and highlighting traditionally overlooked journals with low citations as destination publications (pull), we hope we can change the conversation from ‘Is this a good journal?’ to ‘Does this research matter?’. And we think reframing the context like ScienceOpen is doing is an important first step.