Bibliometric data in rankings: what it captures and what it misses

What bibliometric data actually is

Bibliometric data is the quantitative study of academic publications and their citations. The major databases—Scopus, Web of Science, and Google Scholar—index millions of journal articles, conference papers, and other scholarly outputs, recording who published what, where, and how often each publication has been cited by subsequent work. This data feeds into research-related ranking indicators such as citations per faculty, publications per faculty, the proportion of highly cited papers, and the h-index.

The intuitive appeal of bibliometric data is strong. Unlike reputation surveys, which measure subjective perceptions, citation counts appear to provide objective, verifiable evidence of research impact. A paper that has been cited a thousand times has clearly influenced other researchers. The problem is not that bibliometric data is useless—it is genuinely informative—but that it is incomplete in ways that are not always obvious to ranking users.

Coverage gaps: what the databases miss

The most important limitation of bibliometric data is coverage. No database indexes every scholarly publication. Scopus and Web of Science, the two most commonly used in rankings, are selective databases that include a subset of the world's journals, with a strong bias toward English-language publications from major commercial and society publishers. Research published in smaller journals, in languages other than English, in books or book chapters, in conference proceedings from smaller meetings, or in non-traditional formats is underrepresented or entirely absent.

The consequences of incomplete coverage are not evenly distributed. Universities in non-English-speaking countries, institutions that emphasize locally relevant research, and disciplines where book publication or creative practice is the primary scholarly output are systematically disadvantaged. A university that does excellent research on regional public health issues and publishes in the local language may have near-zero citation counts in international databases, not because the research is low quality, but because it is invisible to the measurement system.

Citation practices across disciplines

Citation practices vary enormously across disciplines, yet rankings often compare citation counts across fields as if they were equivalent. In molecular biology, a typical paper might cite 40 to 50 references. In mathematics, a paper might cite 10 to 15. This is not because biologists are more thorough; it is because the research culture, the structure of the literature, and the norms of attribution differ. An average biology paper will naturally accumulate more citations than an average mathematics paper, regardless of relative quality.

Citation half-lives also vary by field. In fast-moving disciplines, papers are cited quickly and then superseded. In slower-moving disciplines, papers may accumulate citations over decades. A ranking that measures citations over a short window, such as five years, will favor fast-moving fields. A ranking that measures citations over a longer window will favor fields where knowledge accumulates more slowly. Neither choice is neutral with respect to discipline.

What users should know and do

As a ranking user, you should understand that bibliometric indicators are approximations, not direct measures, of research quality and impact. When evaluating a university's research strength, check whether the ranking uses field-normalized citation indicators, which adjust for differences in citation practices across disciplines. Check the time window over which citations are measured and consider whether it is appropriate for the fields you care about. Look for rankings that use multiple bibliometric indicators rather than a single metric, as different indicators capture different aspects of research performance.

Most importantly, use bibliometric data as one input among many in your assessment of research environments. If you are a prospective graduate student, visit department websites and read recent publications from faculty you might work with. Look beyond the numbers to the actual research being produced. Citation counts can tell you that research has been influential, but they cannot tell you whether it is interesting, rigorous, or relevant to your own intellectual interests. That judgment requires reading the work itself.

The appropriate response to the limitations of bibliometric data is not to abandon it, but to use it with understanding. Bibliometric indicators are powerful tools when their limitations are known and their results are interpreted within appropriate disciplinary and linguistic contexts. A ranking-literate user reads citation data as one signal among many—valuable, but incomplete. The richest assessment of research strength will always require reading the research itself, not just counting how often it has been cited.

The bibliometric revolution in rankings has made research output more visible than ever before. That visibility is valuable, but it comes with the responsibility to understand what the numbers leave out. A university that understands its own citation profile, including its limitations, is better positioned to tell its full research story than one that simply reports its h-index.