In the 10+ NIH study sections that I attended in my past life as an academic, one failure mode that repeatedly arises is the False Negative due to the Curse of Experience. The Reviewers in the Study Section, being experts in their field, are so used to thinking about problems in a particular way that they instinctively and subconsciously mistrust any grant application that strongly challenges their worldview.
The review procedure that NIH likes to think is true is that of an unbiased reviewer, reading through each application carefully, noting down the strengths and weaknesses of the application, before coming up with an overall Impact/Priority score. In practice, however, most reviewers will browse the grant application’s title, PI, and Project Summary, come up with a preliminary Impact/Priority score in their minds, and then look for confirming evidence of the good or bad score in the application. The preliminary score can be adjusted by a point or two based on the quality of the data presented, but it’s rare for a grant to be written so well or so poorly as to completely upend the initial judgement.
The startup venture fundraising arena is similar to NIH study sections, in that many VC firms’ partners and principals have in-built biases on how attractive particular types of startups are. In some sense, every VC wants to invest in something new, but not too new. And perhaps rightly so, for General Partners at VC firms have a fiduciary duty and a financial vested interest to maximize the return of the fund. One of the highly limited resources that the GPs need to manage is their team’s time and bandwidth, in terms of performing market and technical diligence.
The saving grace for startup venture fundraising though, is that there are many different VC groups, and all a founder needs is one “Yes.” This is something the NIH could learn from. Rather than taking a vote of Impact/Priority score and taking the average, which typically is driven by loud (and conservative) voices, the NIH could consider drastically reducing the size of Study Section panels, and then giving each panel member a “Golden Ticket” to fund one innovative, high-risk grant. During my experiences as a Study Section member, I remember a number of times where I go to bat for a truly innovative grant, repeatedly raising points in favor of the grant to try to drive the score to a 2, but voting range often ends up 2 to 5, and the grant ends up with a mediocre score that’s not fundable.
At the same time, it is important for VC firms to maintain independent evaluation processes, rather than be swayed by “industry leading VCs.” Fortunately, the free-market system provides financial rewards to VCs who are able to buck convention and find the unlikely winners.
Dave Zhang's Startup Blog 