John Ioannidis doesn't mind stirring the pot. He is famous in the scientific community for very candidly explaining why scientists are getting many things wrong. He has gone so far as to claim that perhaps 90% of the studies
in medical journals are incorrect, and he has the research
to back it up.
A possible explanation for this is publication bias
, the tendency of scientists to submit, and/or journals to accept, positive results (i.e., data which supports a particular hypothesis and is thus "exciting") instead of negative results (i.e., data which essentially says, "Nothing interesting happened here.") As a consequence, journals are crammed full of studies which are never replicated. Moreover, follow-up studies which fail to replicate previous ones often are not published.Writing
in The Atlantic
, David Freedman summarizes the problem well:
Imagine, though, that five different research teams test an interesting
theory that's making the rounds, and four of the groups correctly prove
the idea false, while the one less cautious group incorrectly "proves"
it true through some combination of error, fluke, and clever selection
of data. Guess whose findings your doctor ends up reading about in the
journal, and you end up hearing about on the evening news?
Other explanations include poor or biased experimental design, emphasizing statistical significance over biological relevance, or -- in extreme cases -- outright fraud.
As if all of that wasn't discouraging enough, Dr. Ioannidis has just struck again. This time, he has shown that America's best scientists aren't necessarily the ones to receive funding.
To arrive at this conclusion, he and his co-author Joshua Nicholson examined the most highly cited papers (1,000+ citations) written by U.S.-based biomedical scientists. A random sample of these papers showed that 60% of these exceptional scientists were not
being funded currently by the National Institutes of Health (NIH) as principal investigators (i.e., "lead scientists").
Next, they examined professors in "study sections." These are groups of scientists who have been awarded NIH grants; because of their expertise, they are invited by the NIH to determine which of their colleagues also deserve NIH funding. As expected, most (83%) of study section scientists were currently receiving funds from the NIH.
But, here's the rub: How many of the study section scientists had written highly cited papers? A mere 0.8%. In other words, the scientists deciding which of their colleagues deserve funding are not themselves exceptional scientists. (They may be good scientists, but they aren't the best.) They also tend to award funding to research that is similar to their own.
Combined with the fact that a whopping 60% of exceptional scientists were not
receiving NIH funding, Nicholson and Ioannidis lament that "not only do the most highly cited authors not get funded, but worse, those who influence the funding process are not among those who drive the scientific literature."
They believe that the current funding system has a built-in conflict of interest. Though the NIH is supposed to award funding based solely on merit, that clearly is not always happening. Why? Sometimes it is for benign reasons, such as scientists leaving academia or being in graduate school at the time of their high-impact publication.
But there is also a more malevolent reason, which the authors possibly hint at, but don't explicitly state: Study section scientists do not have a strong incentive to fund exceptional, creative colleagues. Why? Because, very likely, they are also competitors.
Nicholson and Ioannidis propose several solutions, one of which is essentially automatic funding for the truly exceptional scientists. Perhaps this could break the perception that the NIH is rewarding conformity over ingenuity.Source
: Joshua M. Nicholson & John P. A. Ioannidis. "Research grants: Conform and be funded." Nature 492, 34-36 (06 December 2012). doi:10.1038/492034aImage: Pile of Money via Shutterstock