About Journal Rankings

Why journal rankings should not be used when assessing the quality of research

Journal rankings are often used when assessing the quality of research. However, instead of using a ranking as a complement to a thorough reading of the articles, it is sometimes used as a substitute.

Why are journal rankings used?

The short answer to this question can be explained by signaling theory: if an article appears in a highly ranked journal, it is very likely that the research described in the article is of high quality. Obviously, there is not a perfect correlation between a journal’s esteem and the quality of the articles that appear in the journal, but it is cost-effective to rely on the journal’s reputation when evaluating an article’s quality.

Since signaling theory plays a central role in several areas in economics, it should not come as a surprise that journal rankings are widely used when assessing research quality in economics.

Why journal rankings should not be used

As stated above, there is not a perfect correlation between a journal’s esteem and the quality of the articles that appear in the journal. Thus, high-quality articles may appear not only in highly ranked journals but also in journals of lower rank. Therefore, the only way to determine whether a particular article in a journal is of high quality is to actually the read article.

The San Francisco Declaration on Research Assessment (DORA) states that there is a “need to assess research on its own merits rather than on the basis of the journal in which the research is published”, where “the fundamental philosophy behind DORA is that research should be assessed on its own qualities and merits rather than on the basis of surrogate measures of quality, such as the journal in which the research is published” (see https://www.nwo.nl/en/dora).

When journal rankings play a central role in assessing research quality, which is true in economics, researchers tend to formulate their research questions and choose methods so that they maximize the probability of having articles published in the most esteemed journals according to some journal ranking. Obviously, this is detrimental to research and leads to a suboptimal allocation of research resources.

Assessing the quality of an article

Even when an article is actually read by an evaluator, it can still be difficult to assess the quality of the article since one may be affected by the journal’s status, as suggested by the Music Lab experiment (see Salganik, Dodds and Watts in Science 2006).

Specifically, the participants in the experiment listened to, rated, and, if they wished to, downloaded songs by musicians they had never heard of. One group of participants did not receive information about the popularity of the songs in the form of download statistics, whereas this information was provided to the participants in eight other groups, or “worlds,” in the experiment. The purpose of this design was for the former group of participants to determine the quality of the songs, while the participants in the different “worlds” determine the success of the songs, allowing for social influence among participants.

Salganik et al. (2006) found that the success of a song was only partly determined by its quality. In fact, the songs that became popular were different in the different “worlds.” A remarkable example was the song Lockdown. Regarding its quality, the song was ranked 26th of 48 songs. Nevertheless, in one “world,” the same song was ranked 1st, whereas in another “world,” it was ranked 37th.

In a figurative sense, the results of the Music Lab experiment indicate that the perceived status of an article can be strongly affected by the journal’s reputation. However, this problem could be circumvented by simply shifting “publication away from the current fixed format journals and towards an open source arXiv or PLOS ONE format” (p. 55 in Heckman and Moktan NBER No. 25093).

Last but not least, here is an excellent article (in Swedish) on the partly detrimental incentive structure in economics: "Bör alla nationalekonomer stöpas i samma topp-fem form?" (Jonung, Henrekson and Lundahl in Ekonomisk Debatt 2024).