The destructive myopia of the NIH study on grant funding and race

Last week Science published a paper describing the results of an NIH-sponsored investigation into the impact of a scientist’s race on the probability of that their grants will be funded.

The findings were striking:

After controlling for the applicant’s educational background, country of origin, training, previous research awards, publication record, and employer characteristics, we find that black applicants remain 10 percentage points less likely than whites to be awarded NIH research funding.

This result has led to some degree of hand-wringing and pledges to act from the NIH, along with calls to stamp out presumed racism in the peer review process from prominent editorial pages and Congress (and here).

I have no reason to doubt the data presented in the paper. But the interpretation by the authors, media and others are alarmist and incomplete. Rather than exposing lingering racism in the NIH peer review process, I think the data paint a fairly accurate picture of the current state of efforts to increase minority representation in the sciences – highlighting remaining challenges, but also noteworthy accomplishments that the paper and discussions about it have completely ignored.

First, let’s look at the data. The key finding is shown below (I’ve plotted data from Table S1 of the paper. For those unfamiliar with the NIH peer review process, all applications are reviewed by 3 scientists who assign a preliminary score – in the case of these grants from 100 to 900, with 100 being the best – grants with a preliminary score above a threshold are “triaged”, the remainder are discussed with the larger panel of reviewers, after which final scores are recorded):

The effect is undeniably strong: 60 percent of R01 (the standard research grants that support most labs) applications from black PIs are triaged, compared to 40% for white applicants, and far fewer black PIs get scores in the coveted 100-150 range where grants are likely to get funded.

But a few things are worth noting. First, of the applications that are discussed in study section (i.e. not triaged), a roughly equal proportion of applications from black and white PIs get scores between 100-150.

This detail should not be ignored. NIH funding is intensely competitive. Fewer than 10% of all applicants get this kind of score. Far more than just a passing grade, a score better than 150 reflects shared enthusiasm amongst the reviewers and other panelists for the project and the investigator – that they think the proposed work is technically outstanding, innovative and important, and that they have a high degree of confidence that the applicant will accomplish what they have set out.

While the total number of black PIs receiving this kind of acclaim for their work remains smaller than ideal, these results highlight the growing number of tremendously talented and accomplished black researchers rising through the ranks of American science, and serves as a testament to these individuals’ intellect, creativity and hard work. We should celebrate the remarkable success in diversifying American science that their accomplishments represent.

This is important to point out because people reading the paper and most of the discussions surrounding will get the incorrect impression that black scientists face an almost hopeless battle to get funded. And if this misconception were to spread unchecked, it could easily become a self-fulfilling prophesy, with the best students and postdocs reluctant to join labs whose prospects for funding – and thus success – they believe to be low.

Of course the success of some black applicants does not mean that NIH peer review do not give worse scores to grants whose PIs are black than they would to the same proposal from a white scientist. The NIH is (appropriately) doing experiments to investigate this possibility, but, in my service on study sections and other review panels I have never seen anything that suggests race is factoring in to people’s decisions in any way. The NIH peer review system has its flaws, but reviewers take the meritocratic ideal seriously and really do try to be fair. Furthermore, most researchers I know share the goal of diversifying the field, and seem unlikely players in the kind of pervasive racism it would take to explain the observed differences in outcome.

I am not arguing that there is no racism in science. I just don’t think it manifests itself in the grant review process. Consider, for comparison, the success of women in the granting process. Much has been written about the challenges women face in building successful scientific careers, and few would doubt that misogyny persists in the biomedical sciences. But, as researchers using the same database of NIH grant applications that led to the race results recently concluded in a far less high-profile recent study, success rates for men and women were not significantly different in most award programs. While obviously gender and race are not equivalent, I think this argues that the the NIH peer review process is actually fairly equitable, at least in the narrow sense of scoring grant applications.

So, if I don’t believe peer review is pervasively racist, but I believe the data in the paper, I have to believe instead that NIH study sections find that grant applications from black scientists are – on average – marginally less impressive than those of their comparably experienced and accomplished white colleagues. But should this really be that surprising? To expect perfect equity, you have to assume that aspiring black and white scientists have – again, on average – had equally good educational and training opportunities, and that the best and the brightest black students are equally likely to pursue careers in science as their white peers – two things no reasonable person would argue are true. Indeed it is precisely because of these problems that the NIH has put extensive resources into encouraging minorities to enter science and to promote their careers. I strongly support these efforts, but the NIH can not magically erase the effects of disparities in education and research opportunities. And unless you think these efforts have been universally successful, the results presented in this paper should not be surprising.

This is why, rather than looking at the data presented by Ginter et al. as a shocking expose about racism in biomedical science, we should instead look at it as a measure of the progress we’ve made towards diversifying American science: revealing clear remaining challenges that must be addressed, but also highlighting the tremendous accomplishments of the growing number of black and other traditionally underrepresented biomedical scientists.

And I hope it’s not too late to change the message we take from this work. The most disheartening statistic in the paper is not the success rate of black grant applicants, but the fact that there are so few of them (only 1.4% of applications were from black PIs). Rather than discouraging aspiring black scientists by portraying a field filled with insurmountable obstacles, we should emphasize that biomedical science offers them the opportunity to be judged not by the color of their skin, but by the content of their R01 application.

This entry was posted in politics, race, science, science and culture. Bookmark the permalink. Both comments and trackbacks are currently closed.

9 Comments

  1. Tom Turner
    Posted August 26, 2011 at 8:08 am | Permalink

    Thanks for your analysis, Mike. I was thinking along similar lines when I saw the paper, though less clearly.

  2. Posted August 26, 2011 at 11:37 am | Permalink

    Interesting analysis. Particularly since you choose to dismiss the triage as if it is nothing- to paraphrase a feminist mantra, equality will arise when the black PIs have only to do as well as the least accomplished white PIs.

    You ignore the fact that black PIs had to resubmit one more time, on average, to reach these stellar scores.

    You ignore the grey zone of funding exceptions (from this rough analysis, the second bin 151-200 is the closest we can get to this qualitative bin)

    So when you look at the very top of the most excellent proposals, perhaps your analysis stands. But then what? A sharp drop off for the run of the mill “good” black PIs?

  3. Michael Eisen
    Posted August 26, 2011 at 12:09 pm | Permalink

    I’m not sure why you said I dismiss the triage numbers – or the 151-200 scoring range. I just felt that it was important not to just emphasize that there are black PIs whose applications get not just grudging acceptance but actual enthusiasm from reviewers to counter the the negative side of the data which I think gives the false impression of a peer review process whose decks are stacked against black PIs.

    The numbers clearly show a rightward skew for applications from black PIs – something that should concern everyone. And, obviously, we need to figure out why this is happening and do our best to correct it. But I’m shocked at how many people leapt to the immediate conclusion that the peer review system penalizes applications from black PIs when we know that black scientists face all sorts of other obstacles that both discourage them from entering the field in the first place and make it more difficult for them once they are here. I just felt it was pretty naive on the NIHs part to expect anything different – as if they thought the things they were doing to promote the careers of black scientists had actually solved all the problems they face. And then to look at the data and cry racism is just making the problem even worse by both discouraging black scientists from joining the field and making it harder for them to recruit people once there here.

    Anyway – I think there’s a lot more to say about this – and I’ve downloaded and am looking at the raw data. But my point was absolutely not that there is a deficiency of good black PIs, but that the NIH’s apparent thinking on this matter is at once naive and self-defeating.

  4. BikeMonkey
    Posted August 26, 2011 at 12:57 pm | Permalink

    Ping chaps.

  5. Posted August 26, 2011 at 1:02 pm | Permalink

    I just felt that it was important not to just emphasize that there are black PIs whose applications get not just grudging acceptance but actual enthusiasm from reviewers to counter the the negative side of the data which I think gives the false impression of a peer review process whose decks are stacked against black PIs.

    In other news, we have a black president. Therefore, racism is dead.

  6. becca
    Posted August 26, 2011 at 2:33 pm | Permalink

    First- a point of clarification- where did the second graph you posted come from? Why does it look so different from the first graph?

    Second, a point of agreement. Blacks make up 10.2% of the general population, and 2.9% of the full time med school faculty. So yes, they have a LOT of obstacles when it comes to entering the field. This is a huge problem, and a generally pretty well acknowledged one. If you want to claim that problem is bigger than anything at the NIH investigator grant level, you won’t get any arguments from me.

    Third, a point of strong disagreement. The fact there are other problems does not imply there are no problems in NIH grant review. Blacks make up 2.9% of the med school faculty, but 1.2% of the NIH PIs.
    In this climate, problems in getting grants *are* problems “staying in the profession”.

    The whole point of the Ginther study was to *control* for as many characteristics of the applicants as possible. The black applicants that got triaged were no worse than the white applicants that got triaged. There are two basic (not mutually exclusive) possibilities- bias in how the proposals are evaluated, or poorer quality proposals from the black applicants. At present, we don’t have the information that allows us to distinguish between these possibilities.

    I don’t see how any point you’ve made actually *demonstrates* that it was the later and not the former.

    Before you complain about people leaping to “cry racism” how about you question why you yourself leapt to the conclusion the system must be fair?

  7. Michael Eisen
    Posted August 26, 2011 at 7:23 pm | Permalink

    becca – the second graph is the percentages of scored applications – it removes the triaged ones

  8. Michael Eisen
    Posted August 26, 2011 at 7:36 pm | Permalink

    becca again – I understand that they controlled for a whole host of covariates (publication record, grant history) – which is why I said that (bias aside) the data suggest that reviewers think grant applications from black PIs are not as strong on average. Note – in contrast to what Drug Monkey implies – I do not think that a lower score on a grant means that someone is necessarily a poorer scientist, nor do I think that the population of black scientists is intrinsically weaker than anyone else. But successful grantwriting requires more than just raw intelligence and scientific acumen – and it is in precisely some of these indirect acquired skills (writing) and intangibles (networking, mentoring) etc… where the cumulative effects of weaker educational and training opportunities manifest themselves.

  9. Sara
    Posted March 28, 2014 at 7:07 am | Permalink

    NIH “review” is laughable. It’s all who you know, who’s due, who owes who, names, same places, same faces, etc…rigged, and a dog/pony show for sheeple and the delusionally optimistic who “think” their project is going to truly be objectively review for true merit. The reviewers mainly live in the world of make believe (academia) and do not have real world knowledge of how things really work. They want elephants through key holes, walking on water, rabbits out of hats and mansions built all for $1M. Of course, all withing 6months of the start date. Delusional, duped, and in denial.

5 Trackbacks