How many citations are actually a lot of citations?

In a previous blog post, I suggested to my younger colleagues that while they should not care so much about the impact factor of the journals they published in (as long as these journals are well-read in their respective fields of research), they should care quite a lot about these papers being cited, and cited by others not self-cited!

A few months ago, I was listening to the introductory talk of for a prestigious award from our national organization when one statement hit me: a physicist with 2000 or more citations is part of the 1% most cited physicists worldwide. There might have been a bit more to that statement but let’s work with it.

In fact, if you search for highly cited research on Google or any of your favorite search engines, this question (and many related ones) is the subject of intense research in itself. It seems, we human like to be able to put a number and rank on things. We also like to establish a hierarchy among things, even human being. 😉

So according to Wikipedia, from January 2000 to February 2010, the top 1% researchers in physics had about 2073 citations or better, or about slightly fewer than 210 citations per year over a 10-year period. That threshold changes with the field of research, of course, and can be much lower for other fields.

My first impression was that this seems low. However, as one digs deeper, one notices that most published manuscripts are not cited at all…Nature had a nice piece in 2014 about the TOP 100 papers. The infographics below is part of the article and is quite telling.

nature_top_100_papers_infographicV2_30.10.14

 

First, almost 44% of all published manuscripts are never cited. If you have even 1 citations for a manuscript you are already (almost!) in the top half (top 55.8%). With 10 or more citations, your work is now in the top 24% of the most cited work worldwide; this increased to the top 1.8% as you reach 100 or more citations. Main take home message: the average citation per manuscript is clearly below 10!

I also found a great blog post by Scott Weingart, which by restricting his analysis to a single journal, Scientometrics, got very similar results, with 50% of all published papers in that journal having fewer than 4 citations, 70% fewer than 7.

Out of curiosity, I decided to look at my own publications. Two tools are available Google Scholar and Thomson Reuters Web of Sciences. These data are presented in the figure below. As expected the number of manuscripts that gather over 100 citations are few. In my case it hovers between 2% to 5% depending on the tool used; both gives over 60% for 10 or more citations: 2 to 5 manuscripts out of a 100 get to be in the top 1.8% most cited manuscripts and 60 out of the same 100 are in the top quarter. Based on the discussion above, I suppose this is a good sign…

CItation-Fraction

 

Still in Nature, there was yet another piece about the TOP 1% in science. Here is it claimed that fewer than 1% of all researchers have published consistently every year between 1996 and 2011, but those few who have commanded a “market share”  of (i.e. are authors on) 41% of all published manuscripts in the same period (original manuscript in PLOS ONE).

There is, of course, Thomson Reuters that has its own version of the “World’s Most Influential Scientific Mind” for a given year (e.g. 2014) and highly cited researchers. Here “hot papers” are defined by being in the top 0.1% (you’ve read correctly) by citations for their field of research and “influential researchers” are those having the largest numbers of these “hot papers” (generally 15+). Before you ask, no I am not listed in the 3500 or so researchers described in there…

Now having said all that, more analysis of all of these “metrics” seems to indicate that there is only a weak correlation between the top 1% of highly cited researchers and Nobel Prize winners. As noted by the first Nature article cited above, many of the very highly cited papers are about tools or methods rather than fundamental scientific discoveries made with these tools or methods…

Here is a quick one that was part of a discussion among friends: two researchers have each a paper with 200+ citations. For the purpose of this blog post let’s say exactly 200 citations. One is for a paper published in what is considered a top journal, let’s say Science or Nature. The other for a manuscript published in a low impact factor journal (5-year IF of 2). Which of these two papers have had:

  1. The greatest impact on science?
  2. On their field of research?

Looking forward to reading you!

Posted on November 19, 2015, in Research and Academia and tagged , , , , , , . Bookmark the permalink. 4 Comments.

  1. Nice article Luc. A minor observation is that those highly cited papers tend to be slanted towards biological sciences, which also tend to have a fairly high number of publications. Putting individual statistics aside, I wonder if citations is loosely correlated to the number of sub-disciplines.

    Defining an ‘expert’ is remarkably different from someone who has a high H-index, # of citations, and other publication metrics, and more to do with things like invited lectures, publications in reports, books, guidance documents, and such which are may not be subjected to indexing.

    I whole heartedly agree that, for younger researchers, quantity is important, without forsaking quality. And things like blog-posts, comments, etc all count! Later in ones academic career one can be more picky about where to publish.

    • My experience is that shooting for number only is not the way to go. I am not saying that number of publication is not important since both for promotion and the purpose of obtaining grants, you do need a certain number of them. However, if your publications are not cited, you can have many hundreds it does not matter: you are having no impact on your field.

      In grants committees, my experience is that after a few publications per year, it does not matter so much if you have 18, 22 or 25 in the last 5 years. It will be held against you if you have only 2 or 3 however – unless it is 2-3 Science or Nature (Of course of you have 50+ all in good journals, you are not only a “middle” author on these and you can show that your papers have impact – you will certainly get praised).

      I would also say that you have to pick carefully where you published. And here you are right your “forum” depends a lot on the path you are following: top research journals in your field vs more applied (clinical for us) journals vs reports and so on. It also depends I guess if you need to show social impact, academic impact, clinical impact …

      There is no simple formula for sure 😉

  2. jasonmcdermott873738820

    Very interesting post. It seems like a small number (2000 being the 1%) but I don’t have any papers close to those citation numbers. A couple of years ago I posted a fairly simple analysis of my own citation record. But I thought it would be interesting to see how my papers performed in relation to the impact factor of the journal they were in. Most did pretty well, getting more than expected by the journal. Anyway, I thought that would be an interesting addition to your own self analysis (though it’s complicated by the fact that the impact factor of journals varies over time). Here’s the post if you’re interested: http://jasonya.com/wp/publication-impact-factor-analysis/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: