Leaders in Pharmaceutical Business Intelligence (LPBI) Group

Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method that makes novel use of the co-citation network of each article to field-normalize the number of citations it has received. The resulting Relative Citation Ratio is article-level and field-independent, and provides an alternative to the invalid practice of using Journal Impact Factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010, and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research, and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov.

In the current highly competitive pursuit of research positions and funding support (CouzinFrankel, 2013), faculty hiring committees and grant review panels must make difficult predictions about the likelihood of future scientific success. Traditionally, these judgments have largely depended on recommendations by peers, informal interactions, and other subjective criteria. In recent years, decisionmakers have increasingly turned to numerical approaches such as counting first or corresponding author publications, using the impact factor of the journals in which those publications appear, and computing Hirsch or H-index (Hirsch, 2005). The widespread adoption of these metrics, and the recognition that they are inadequate (Seglen, 1997; Anon, 2005, Anon, 2013), highlight the ongoing need for alternative methods that can provide effectively normalized and reliable data-driven input to administrative decision-making, both as a means of sorting through large pools of qualified candidates, and as a way to help combat implicit bias. A return to purely subjective evaluation with its attendant risk of partiality is neither desirable nor practical, and the use of metrics that are of limited value in decision-making is widespread and growing (Pulverer, 2013). The need for useful metrics is particularly pressing for funding agencies making policy decisions based upon the evaluation of large portfolios that often encompass diverse areas of science. Though each of the above mentioned methods of quantitation has strengths, accompanying weaknesses limit their utility. Counting first or corresponding author publications does on some level reflect the extent of a scientist’s contribution to their field, but it has the unavoidable effect of privileging quantity over quality, and may undervalue collaborative science (Stallings et al., 2013). Journal impact factor (JIF) was for a time seen as a valuable indicator of scientific quality because it serves as a convenient, and not wholly inaccurate, proxy for expert opinion (Garfield, 2006). However, its blanket use also camouflages large differences in the influence of individual papers. This is because impact factor is calculated as the average number of times articles published over a two-year period in a given journal are cited; in reality, citations follow a log-normal rather than a Gaussian distribution (Price, 1976; Wang et al., 2013).

Figure 1. Properties of co-citation networks. (a) Schematic of a co-citation network. The Reference Article (RA) (red, middle row) cites previous papers from the literature (orange, bottom row); subsequent papers cite the RA (blue, top row). The co-citation network is the set of papers that appear alongside the article in the subsequent citing papers (green, middle row). The Field Citation Rate is calculated as the mean of the latter articles’ journal citation rates. (b) Growth of co-citation networks over time. Three RAs published in 2006 (red dots) were cited 5 (top row), 9 (middle row), or 31 times (bottom row) by 2011. Three intervals were chosen to illustrate the growth of the corresponding co-citation networks: 2006-2007, 2006-2009, and 2006-2011 (the first, second, and third columns, respectively). Each article in one of the three co-citation networks is shown as a separate green dot; the edges (connections between dots) indicates their presence together in the same reference list. (c) Cluster algorithm-based content analysis of the 215 papers in the co-citation network of a sample reference article (RA; panel b, bottom network series) identified a changing pattern of relevance to different sub-disciplines over time. This RA described the identification of new peptides of possible clinical utility due to their similarity to known conotoxins. Papers in the co-citation network of this RA focused on: (1) α- conotoxin mechanisms of action; (2) structure and evolution of conotoxins; (3) cyclotide biochemistry; (4) conotoxin phylogenetics; and (5) identification and synthesis of lantibiotics. (d) Growth of an article’s co-citation network is proportional to the number of times it has been cited. Each point is the average network size of 1000 randomly chosen papers with between 1 and 100 citations (error bars represent the standard error of the mean). Each paper is only counted once, even if it is co-cited with the article of interest multiple times. An average of 17.8 new papers is added to the co-citation network for each additional citation. This suggests substantial duplication of articles within a co-citation network, since on average 32.4 papers (median of 30) are referenced in each citing article.

Figure 2. Algorithm for calculating the Relative Citation Ratio. (a) Article Citation Rate (ACR) is calculated as the total citations divided by the number of years excluding the calendar year of publication (Supplemental Equation 1), when few, if any, citations accrue (Supplemental Figure 2). (b) Generate an expectation for article citation rates based on a preselected benchmark group, by regressing the ACR of the benchmark papers onto their FCRs (Supplemental Equations 3, 4), one regression each publication year. The graphed examples were sampled from a random distribution for illustrative purposes. (c) The coefficients from each year’s regression equation transforms the Field Citation Rates of papers published in the same year into Expected Citation Rates (Supplemental Equation 5). Each paper’s RCR is its ACR/ECR ratio. A portfolio’s RCR is simply the average of the individual articles’ RCRs (Supplemental Equation 6). (d) Boxand whisker plots of 88,835 NIH-funded papers (published between 2003 and 2010), summarizing their Article Citation Rate, Journal Impact Factor (matched to the article’s year of publication), and Field Citation Rate. Boxes show the 25th-75th percentiles with a line at the median; whiskers extend to the 10th and 90th percentiles.

Having chosen our comparison group, the next step was to decide how to calculate the values that numerically represent the co-citation network of each RA. The most obvious choice, averaging the citation rates of articles in the co-citation network, would also be highly vulnerable to finite number effects. We therefore chose to average the citation rates of the journals represented by the collection of articles in each co-citation network. If a journal was represented twice, its journal citation rate (JCR) was added twice when calculating the average JCR. For reasons of algorithmic parsimony we used the JCRs for the year each article in the co-citation network was published; a different choice at this step would be expected to have little if any effect, since almost all JCRs are quite stable over time (Supplemental Figure 1; Supplemental Table 1). Since a co-citation network can be reasonably thought to correspond with an RA’s area of science, the average of all JCRs in a given network can be redefined as that RA’s field citation rate (FCR).

Using this method (Figure 2a-c; Supplemental Figure 2; Supplemental Equations 1 and 2), we calculated FCRs for 35,837 papers published in 2009 by NIH grant recipients, specifically those who received R01 awards, the standard mechanism used by NIH to fund investigator-initiated research. We also calculated what the FCR would be if it were instead based on citing or cited networks. It is generally accepted that, whereas practitioners in the same field exhibit at least some variation in citation behavior, much broader variation exists among authors in different fields. The more closely a method of field definition approaches maximal separation of between field and within-field citation behaviors, the lower its expected variance in citations per year (CPY). FCRs based on co-citation networks exhibited lower variance than those based on cited or citing networks (Table 1). Interestingly, a larger analysis of the 88,835 papers published by investigators with continuous R01 funding between 2003 and 2010 shows that FCRs also display less variance than either ACRs (p < 10-4, F-test for unequal variance) or JIFs (p < 10-4, F-test for unequal variance, Figure 2d, Table 1), confirming that co-citation networks are better at defining an article’s field than its journal of publication.

For the work presented here, we chose as a benchmark the full set of 311,497 RAs published from 2002 through 2012 by NIH-R01 awardees. To measure the degree of correspondence between our method and expert opinion, we compared RCRs generated by benchmarking ACR/FCR values against this standard to three independent sets of post-publication evaluations by subject matter experts (details in Supplemental Information). We compared RCR with expert rankings for 2193 articles published in 2009 and evaluated by Faculty of 1000 members (Figure 3a), as well as rankings of 430 Howard Hughes Medical Institute- or NIH-funded articles published between 2005 and 2011 and evaluated in a study conducted by the Science and Technology Policy Institute (STPI, Figure 3b), and finally, 290 articles published in 2009 by extramurally funded NIH investigators and evaluated by NIH intramural investigators in a study of our own design (Figure 3c; Supplemental Figs. 5-7). All three approaches demonstrate that RCR values are well correlated with reviewers’ judgments. We asked experts in the latter study to provide, in addition to an overall score, scores for several independent subcriteria: likely impact of the research, importance of the question being addressed, robustness of the study, appropriateness of the methods, and human health relevance. Random forest analysis indicated that their scores for likely impact were weighted most heavily in determining their overall evaluation (Supplemental Figure 6).

In addition to correlating with expert opinion, RCR is ranking invariant, which is considered to be a desirable property of bibliometric indicators (Rousseau and Leydesdorff, 2011; Glänzel and Moed, 2012). In short, an indicator is ranking invariant when it is used to place two groups of articles in hierarchical order, and the relative positions in that order do not change when uncited articles are added to each group. The RCR metric is ranking invariant when the same number of uncited articles is added to two groups of equal size (Supplemental Equations 7-9). RCR is also ranking invariant when the same proportion of uncited articles is added to two groups of unequal size (Supplemental Equations 10-11). This demonstrates that the RCR method can be used effectively and safely in evaluating the relative influence of large groups of publications.

Figure 3. Relative Citation Ratios correspond with expert reviewer scores. (a-c) Bubble plots of reviewer scores vs. RCR for three different datasets. Articles are binned by reviewer score; bubble area is proportionate to the number of articles in that bin. (a) F1000 scores for 2193 R01-funded papers published in 2009. Faculty reviewers rated the articles on a scale of one to three (“Good”, “Very Good”, and “Exceptional”, respectively); those scores were summed into a composite F1000 score for each article (Supplemental Figure 3). (b) Reviewer scores of 430 HHMI and NIH-funded papers collected by the Science and Technology Policy Institute. (c) Scores of 290 R01-funded articles reviewed by experts from the NIH Intramural Research Program. Black line, linear regression.

Recent work has improved our theoretical understanding of citation dynamics (Radicchi et al., 2008; Stringer et al., 2010; Wang et al., 2013). However, citation counts are not the primary interest of funding bodies, but rather progress in solving scientific challenges. The NIH particularly values work that ultimately culminates in advances to human health, a process that has historically taken decades (Contopoulos-Ioannidis et al., 2008). Here, too, metrics have facilitated quantitation of the diffusion of knowledge from basic research toward human health studies, by examining the type rather than the count of citing articles (Weber, 2013). Insights into how to accelerate this process will probably come from quantitative analysis. To credit the impact of research that may currently be underappreciated, comprehensive evaluation of funding outputs will need to incorporate metrics that can capture many other outputs, outcomes, and impact, such as the value of innovation, clinical outcomes, new software, patents, and economic activity. As such, the metric described here should not be viewed as a tool to be used as a primary criterion in funding decisions, but as one of several metrics that can provide assistance to decisionmakers at funding agencies or in other situations in which quantitation can be used judiciously to supplement, not substitute for, expert opinion.

Table 1. Variance of Field Citation Rates and Expected Citation Rates using different levels of the citation network for calculations (based on 35,837 R01-funded papers published in 2009).

Network level                             Variance (FCR)                             Variance (ECR)

Impact Factor (RA only)                  33.7                                                      30.8

Cited network                                  16.1                                                        9.5

Citing network                                   6.7                                                         7.8

Co-citation network                           3.4                                                         3.4

Table 2. Summary of investigator-level bibliometric measures and their stability from one 4-year period to the next (PIs with 5 or more articles in each period, except for article count).

Measure                                      Mean                       Median                  Pearson r
(of log-values, ’03- ’06 vs ’07-’10)

Articles per interval (PIs with >0
articles in both 4-yr. intervals)         9.8                        8.0                                0.56

Field citation rate                             7.8                        7.7                                 0.92

Journal citation rate                         6.3                        5.7                                 0.76

Article citation rate                           6.4                        5.3                                 0.67

RCR                                                 1.0                        0.85                              0.61

Stefano Bertuzzi ASCB

What if I told you that nearly 90 percent of the publications which have profoundly influenced the life sciences did not appear in a high-impact factor journal? If you signed the San Francisco Declaration on Research Assessment, you probably aren’t surprised. If you haven’t signed DORA, it may be time for you to reconsider the connection between true breakthrough papers and so-called journal impact factors (JIFs).

Today we received strong evidence that significant scientific impact is not tied to the publishing journal’s JIF score. First results from a new analytical method that the National Institutes of Health (NIH) is calling the Relative Citation Ratio (RCR) reveal that almost 90% of breakthrough papers first appeared in journals with relatively modest journal impact factors. According to the RCR, these papers exerted major influence within their fields yet their impact was overlooked, not because of their irrelevance, but because of the widespread use of the wrong metrics to rank science.

In the initial RCR analysis carried out by NIH, high impact factor journals (JIF ≥ 28) account for only 11% of papers that have high RCR (3 or above). Here is hard evidence for what DORA supporters have been saying since 2012. Using the JIF to credit influential work means overlooking 89% of similarly influential papers published in less prestigious venues.

The RCR is the creation of an NIH working group led by George Santangelo in the Office of the NIH Director. Santangelo has just uploaded an article in the Cold Spring Harbor BioArchive repository describing the RCR metric. I believe that the Santangelo proposal would significantly advance research assessment.

This marks a significant change in my own thinking. I am firmly convinced that no single metric can serve all purposes. There is no silver bullet in research evaluation; qualitative review by experts remains the gold standard for assessment. And yet I would bet that this new metric will gain currency, contributing to a new and better understanding of impact in science. The RCR provides us a new sophisticated analytical tool, which I hope will put another nail into the coffin of the phony metric of the journal impact factor. As I and many others have said many times, the JIF is the wrong way of assessing article-level or, even worse, individual productivity.

So, what is this new RCR metric? The Relative Citation Ratio may seem complicated at first glance but the concept is simple and very clever. Key to the RCR is the concept of co-citation network. In essence, this new metric compares the citations an article receives to a custom-built citation network which is relevant to that particular paper. The relevant network is defined by the entire collection of papers which are referenced in the papers that cite the reference paper. All this constitutes the denominator of the RCR, while the numerator is simply the citations received by the reference article.

The values used to calculate the denominator of the defined citation network are based on the Journal Citation Rate (JCR), which is also used to compute in journal impact factor. But it is important to note that the RCR, besides being based on the co-citation network, places the journal metric, the JCR, NOT at the numerator, but at the denominator. This makes this new RCR a robust article-level metric normalized to the citations in the custom-built field of relevance and to the expected citations that journals receive in that network. The RCR is then normalized to make comparisons easier. To do so the authors use the cohort of NIH R01 funded papers as the benchmark set.

NIH will provide full access to the algorithms and data to calculate the RCR, making this a highly transparent and accessible tool for the whole scientific community. This is a fundamental change in assessment and it is incredibly exciting.

I am not a bibliometrician, so I don’t pretend to have all the skills to evaluate the metric algorithm in detail. But I am familiar with research evaluation and, after reading this paper carefully, I am convinced it adds something important to our toolbox in the thorny field of research assessment. I am reminded by something that the legendary Nobel laureate Renato Dulbecco once told me: Based on the JIF metric not only would Dulbecco have never been awarded the Nobel Prize, but he probably would have never gotten a job, since his landmark papers were published in rather obscure journals.

This is exactly the point underscored by this new RCR analysis. Often highly innovative ideas—new concepts, technologies, or methods— may be of immediate interest only to a very small group of scientists within their highly specialized area. These seemingly arcane advances attract little notice outside that subfield. Yet on the meandering roads of research, an obscure breakthrough with seemingly little relevance to outsiders may reorient the field. What began with a curiosity-driven observation reported in an obscure journal may roll on to become a landmark discovery. I believe the RCR addresses this problem. My concerns about miracle metrics were assuaged by NIH’s careful benchmarking of the RCR, using expert qualitative review of the RCR scored papers that reported strong concordance.

After years of blasting one-size-fits-all metrics, I find myself in the uncomfortable position of cheering for a new one. Yes, the RCR must be road tested further. It must be tried in multiple fields and situations, and modified, if necessary, to address blind spots. And I still hold that qualitative review by experts remains the gold standard for individual assessment. But from this early report by the Santangelo group, I am convinced that here is a metric that reflects how science really evolves in laboratories, scientific meetings, and in obscure journals. It evaluates science by putting discoveries into a meaningful context. I believe that the RCR is a road out of the JIF swamp.

Well done, NIH.

The RCR metric has been made available in an online tool and has already received considerable attention. Stefano Bertuzzi, Executive Director of the American Society for Cell Biology, strongly endorses the metric in a blog post and calls it ‘stunning’ and ‘very clever’. However, does the RCR metric really represent a significant step forward in quantifying scientific impact? Below I will explain why the metric doesn’t live up to expectations.

iCite is a tool to access a dashboard of bibliometrics for papers associated with a portfolio. Users upload the PubMed IDs of articles of interest (from SPIRES or PubMed), optionally grouping them for comparison. iCite then displays the number of articles, articles per year, citations per year, and Relative Citation Ratio (a field-normalized metric that shows the citation impact of one or more articles relative to the average NIH-funded paper). A range of years can be selected, as well as article type (all, or only research articles), and individual articles can be toggled on and off. Users can download a report table with the article-level detail for later use or further visualization.

Launch iCite

Stefano Bertuzzi ASCB

 http://www.ascb.org/nature-of-scientific-impact/

In a simplified form, the idea of the RCR metric can be summarized as follows. To quantify the impact of a publication X, all publications co-cited with publication X are identified. A publication Y is co-cited with publication X if there is another publication in which publications X and Y are both cited. The publications co-cited with publication X are considered to represent the field of publication X. For each publication Y belonging to the field of publication X, a journal impact indicator is calculated, the so-called journal citation rate, which is based on the citations received by all publications that have appeared in the same journal as publication Y. Essentially, the RCR of publication X is obtained by dividing the number of citations received by publication X by the field citation rate of publication X, which is defined as the average journal citation rate of the publications belonging to journal X’s field. By comparing publication X’s number of citations received with its field citation rate, the idea is that a field-normalized indicator of scientific impact is obtained. This enables impact comparisons between publications from different scientific fields.

According to the NIH team, “citation metrics must be article-level, field-normalized in a way that is scalable from small to large portfolios without introducing significant bias at any level, benchmarked to peer performance in order to be interpretable, and correlated with expert opinion. In addition, metrics should be freely accessible and calculated in a transparent way.” The NIH team claims that the RCR metric meets each of these criteria, while other indicators proposed in the bibliometric literature always violate at least one of the criteria. If the NIH team were right, this would represent a major step forward in the development of bibliometric indicators of scientific impact. However, the NIH team significantly overstates the value of the RCR metric.

The most significant weakness of the RCR metric is the lack of a theoretical model for why the metric should provide properly field-normalized statistics. In fact, it is not difficult to cast doubt on the theoretical soundness of the RCR metric. The metric for instance has the highly undesirable property that receiving additional citations may cause the RCR of a publication to decrease rather than increase.

Imagine a situation in which we have two fields, economics and biology, and in which journals in economics all have a journal citation rate of 2 while journals in biology all have a journal citation rate of 8. Consider a publication in economics that has received 5 citations. These citations originate from other economics publications, and these citing publications refer only to economics journals. The field citation rate of our publication of interest then equals 2, and consequently we obtain an RCR of 5 / 2 = 2.5. Now suppose that our publication of interest also starts to receive attention outside economics. A biologist decides to cite it in one of his own publications. Apart from this single economics publication, the biologist refers only to biology journals in his publication. Because biology journals have a much higher journal citation rate than economics journals, the field citation rate of our publication of interest will now increase from 2 to for instance (5 × 2 + 1 × 8) / 6 = 3 (obtained by assuming that 5/6th of the publications co-cited with our publication of interest are in economics and that 1/6th are in biology). The RCR of our publication of interest will then decrease from 5 / 2 = 2.5 to 6 / 3 = 2. This example shows that receiving additional citations may cause a decrease in the RCR of a publication. Especially interdisciplinary citations received from publications in other fields, characterized by different citation practices, are likely to have this effect. Publications may be penalized rather than rewarded for receiving interdisciplinary citations.

Many more comments can be made on the theoretical soundness of the RCR metric. For instance, one could criticize the use of journal citation rates in the calculation of a publication’s field citation rate. If a publication is co-cited with a publication in Science, its field citation rate will depend on the journal citation rate of Science, which in turns depends on the citations received by a highly heterogeneous set of publications, since Science publishes works from many different research areas. It then becomes questionable whether a meaningful field citation rate will be obtained. However, rather than having a further technical discussion on the RCR metric, I will focus on two other claims made by the NIH team.

First, the NIH team claims that “RCR values are well correlated with reviewers’ judgments”. Although the NIH team has put an admirable amount of effort into validating the RCR metric with expert opinion, this claim needs to be assessed critically. The NIH team has performed an extensive analysis of the correlation of RCR values with expert judgments, but it hasn’t performed a comparison with similar correlations obtained for other metrics. Therefore we still don’t know whether the RCR metric correlates more favorably with expert opinion than other metrics do. Given the theoretical problems of the RCR metric, I in fact don’t expect such a favorable outcome.

Second, the NIH team claims that a strength of the RCR metric relative to other metrics is the transparency of its calculation. This is highly contestable. The calculation of the RCR metric as explained above is fairly complex, and this is in fact a simplified version of the actual calculation, which is even more complex. It for instance involves the use of a regression model and a correction for the age of publications. Comparing the RCR metric with other metrics proposed in the bibliometric literature, I would consider transparency to be a weakness rather than a strength of the RCR metric.

Does the RCR metric represent a significant step forward in quantifying scientific impact? Even though the metric is based on some interesting ideas (e.g., the use of co-citations to define the field of a publication), the answer to this question must be negative. The RCR metric doesn’t fulfill the various claims made by the NIH team. Given the questionable theoretical properties of the RCR metric, claiming unbiased field normalization is not justified. Correlation with expert opinion has been investigated, but because no other metrics have been included in the analysis, a proper benchmark is missing. Claiming transparency is problematic given the high complexity of the calculation of the RCR metric.

During recent years, various sophisticated field-normalized indicators have been proposed in the bibliometric literature. Examples include so-called ‘source-normalized’ indicators (exemplified by the SNIP journal impact indicator provided in the Elsevier Scopus database), indicators that perform field normalization based on a large number of algorithmically defined fields (used in the CWTS Leiden Ranking), and an interesting indicator proposed in a recent paper by the Swedish bibliometrician Cristian Colliander. None of these indicators meets all of the criteria suggested by the NIH team, and none of them offers a fully satisfactory solution to the problem of quantifying scientific impact. Yet, I consider these indicators preferable over the RCR metric in terms of both theoretical soundness and transparency of calculation. Given the sometimes contradictory objectives in quantifying scientific impact (e.g., the trade-off between accuracy and transparency), a perfect indicator of scientific impact probably will never be found. However, even when this is taken into account, the RCR metric doesn’t live up to expectations.