lp11 The Peer Review system

Seeking advice from experts before making an important decision is common sense. Indeed it was said that when Christopher Columbus conceived of the idea of reaching China and Japan by sailing west, he submitted a proposal to Queen Isabella of Spain. She sought advise from a panel of three experienced navigators, who unanimously recommended rejection. Later she suddenly changed her mind---apparently prompted by the information provided by her keeper of privy purse that it would cost no more than giving a big party to finance Columbus's trip (see D.J.Boorstein, "The Discoverers", Random House, p. 228). The New World was discovered in consequence.

Large scale peer review as the basis of government funding of scientific research took place after the Second World War. At the time, a common belief was that science made progress by a smooth incremental or cumulative process. Influential people like George Sarton ( The History of Science and the New Humanism, Cambridge University Press, 1937), James B. Conant, and Lord Ernest Rutherford all believed and promoted such a view. That being the case, it seemed reasonable to expect that the advise from a panel of experts or peers is the most reliable way of correctly determining the merit of a piece of proposed research.

This once popular view of cumulative growth of science was later shown to be only partially correct and largely mistaken. Major upheavals or revolutions often took place in between periods of incremental and cumulative developments. Unfortunately, by the time the mistake was broadly recognized and a correct view adopted through the writings of Herbert Butterfield (The Origin of Modern Science 1300-1800, G. Bell and Sons, London, 1949), Rupert Hall (The Scientific Revolution 1500-1800, Longmans, Green and Co,. London, 1954) and Thomas Kuhn (The Structure of Scientific Revolution, Univ. Chicago Press, Chicago, 1962), the peer review system was an established tradition in US government institutions and elsewhere.Worse, in more than one confused mind, peer review has replaced objective evidence as proof of truth.

The peer review systems adopted for government-sponsored research funding can be roughly sorted out into two types. One type was first adopted by the Office of Naval Research (ONR) and the National Science Foundation (NSF); the other type was that introduced by the National Institutes of Health (NIH).

1.The ONR-NSF system

This system places greater responsibility on the individual program managers. A researcher seeking support writes a proposal describing what he/she plans to do and submits it along with the endorsement from the Institution where he/she is employed. The specific program manager covering that field reads the proposal and sends it out to different experts in the field and seeks their opinions. Based on the opinions gathered, the program manager makes the final decision to fund the proposal or to reject it. The good side of this system is that it has flexibility. When the program manager is dedicated and has the needed knowledge, courage and integrity, the system can work well.

The down side of the ONR-NSF system is that if the program manager does not have enough knowledge and self confidence, and/or cares little about the true objective of scientific research, he/she can do a great deal of damage to Science.

Thus in the words of Representative John B. Conlan from Arizona: " ( the system can turn into) an 'Old Boy's System' where program managers rely on trusted friends in the academic community to review their proposals. These friends recommend their friends as reviewers....It is an incestuous buddy system that frequently stifles new ideas and scientific breakthroughs, while carving up the multimillion dollar Federal research and education pie in a monopoly game of grantsmanship..." (from Testimonial on July 22, 1975, as a member of the Subcommittee on Science, Research and Technology of the Committee on Science and Technology, U.S. House of Representatives, 94th Congress, First Session. Quoted from "National Science Foundation Peer Review, Special Oversight Hearings., Publication No. 32, p. 5; U.S. Government Printing Office, Washington D.C.).

2.The NIH system.

NIH is a much larger conglomerate of semi-independent Institutions (e.g., National Cancer Institute, National Heart Institute). Each has its own research laboratories and researchers, constituting what is called "intramural activities". It is the "extramural activity" that oversees funding of research conducted by researchers outside NIH, often in the Universities. This funding is handled by a single NIH institution called the Division of Research Grants. Although there is a second level of peer review by a national advisory council made up by scientists and laymen, the major decision on the funding of research proposal is made by scientists sitting on the Initial Review Groups (IRG) also called Study Sessions. As of 1993, there were a total of 84 study sessions belonging to 16 NIH Institutes.

Research proposals, written by research grant applicants are distributed by the Divisioon of Research Grants to one of the Study Sessions for review. Of those proposals recommended for approval, each of the Study Section members gives (anonymously) a priority score ranging from 100 (highest) to 500( lowest) and the average determines funding or non-funding. Sometimes a difference of a few points determines "life" or "death" of the proposal. Even though each Study Section member (which usually number between 10 and 20 but may be more) gives a numerical score, only two of the members read any one proposal. For this reason, there is no chance for a proposal to be funded, unless those two readers absolutely wish the proposal to be funded; they are the proposals's only advocates. None of the other member really have the detailed knowledge of the proposal to challenge the opinions of the readers, not to mention that they must all cooperate with one another so that proposals read by other members of the Study Section can be accepted or rejected as recommended by their readers.

The power wielded by the two "readers" can be seen from an NIH-distributed document giving advise to grant applicants: "the author of a project proposal must learn all he can about those who will read his proposal and keep these readers in mind constantly as he writes." Since single copies of this advice could be obtained from the Division of Research Grants, NIH (see linked page lp11a), it represents what is considered acceptable and perhaps even desirable by the NIH authority.

There are two ways of looking at this advice. For the great majority of grant applicants working in an area of research where the foundation concepts have been firmly established and acceptable to all, this advice tells you about a system with which one might find fault but perhaps also live.

It is an altogether different story when the applicant works in an area of science where the foundation is far from being unequivocally established, as it is in the case of cell physiology. It is in this situation that the absolute, unchallengeable power given to the "readers" of the Study Sections, with their long four-year tenure, their high visibility, their privileged tradition of recommending their own successors, as well as their own needs for the same NIH money for their own research now being disputed, that the working of the system takes on a significance of truly alarming nature. It is here that the NIH peer review could fall into the same pitfall of generating an "incestuous buddy system" that "stifles new ideas and scientific breakthroughs" as Representative John B. Conlan had described in regard to the NSF system.