Scientific Productivity, Research Funding, Race and Ethnicity

Page 1 Scientific Productivity, Research Funding, Race and Ethnicity J.S. Yang 1,2 , M. W . Vanni er 3 , F. W a ng 4 , Y . Den g 4 , F.R. Ou 4 , J.R. Benne tt 1 , Y. Liu 4 *, G. W an g 1 * 1 VT - W FU School o f Biomedical En gineering and Sciences , V irginia T ech, Bl acksburg, USA 2 School of Mathematical Sciences, Pe king University , Beijing, China 3 Department of Radi ology , University of Chicago, C hicago, USA 4 School of Public Hea lth, China M edical Univ ersity , Shenyang, China * T o whom correspondence should be ad dressed ( YL : cm uliuyang@yahoo.com ; GW : ge - wang@ieee.org ) ABSTRACT In a recent stud y by Gi nther et al ., th e pr obability of r eceiving a U.S. National Institut es of Health (NIH) RO1 award was relate d to the applicant’s race/ethnic ity . T he result s indicate black/African - American appl icants were 10% less likely than white peers to r eceive an award, af ter controlling f o r background and qualifications . It has g enerated a widespread d ebate regardi ng the unfairness of the NIH grant review proc ess and its c orrection. In this paper , the work by Ginther et al . was augm ented by pairing analysis, axiom atically-individualized pr oductivity and norm alized funding success measurement. Although there are racial differences in R01 grant succes s rates, normalized figures of m erit for funding succes s explain the discrep ancy . T he suggested “ l everage points for policy intervention ” are in question and require deeper a nd more thorough investiga tions. Further adjust ments in policies to rem ove racial disparity should be m ade more systematicall y for equal opportunity , rather than being lim ited to the NIH revie w process. 1. BACKGROU ND In a recent study ( D. K. Ginther et al .: “Race, ethni city , and NIH research awards,” Sci ence, 19 August, p . 1015 ) , the probability of recei ving a U.S. N ational Institutes of Heal th (NIH ) RO1 aw ar d w as related to the applicant’ s race/ethnicity . The results indicate black/A frican-American ap plicants were 10% less li kely than whi te peer s to recei ve an award, after controlling for background and qualifications, and further suggest “ leverage points for policy inte rvention ” [1] . These findings ha ve generated a widespread debate regarding the unfairness of the NIH grant revi ew process and its correction . The moral impe rative is cl ear that any hidden racial bias is not to be tolerated, particularly in the NIH funding process. Howev er, the question of whe ther such a racial bi as truly exists requires unbiased , rigoro us and systematic evaluation. NIH director Francis C ollins and Deputy Director Lawrence T ab ak reiterated that the Ginther study reveal ed “ from 2000 to 2006, black ( 1 ) grant applicants were significantly less likely t o receive NIH research funding than were whi te applicants. The gap in success r ates amounted to 10 percentage points, even a ft er cont rolling for educati on, cou ntry of orig in, training, employe r characte ristics, previous res earch awards, and publication r ecord ( 2 ). Their analysis also sho wed a gap o f 4.2 percen tage points for Asians; however , the differences be t ween Asian and white award probabili ties were expl ained by ex clusion of noncitizen s fr om the analysis ” [2]. N I H of ficials admitt ed “ the ga p could also result from ‘insidious’ bias favoring whites in a peer -review system that suppose dly ranks applicatio ns only on scientific merit ” [3]. In a Lette r to Editor of Science, Dr . V oss ex pressed uneasiness about proposals which address implications of the Ginther study [4] . He warned that “ disparity-reduc tion policies represent social experiments with tremendously important consequences, t he effects of which could take decades to identify … much of the racial disparity reported could be attributed to black R01 applicants having half the citation count and one- fi fth as m any last-authored publications as white applicants from similarly ranked in stitutions. Coupl ed with the finding that R 01s we re awarded to highly ranked appl ications irrespec t ive of Page 2 race, this sugge sts that R01 dispa rity is due to l ow er r esearch success among black applican ts rather than to any problems with NIH review ” [4]. In another Letter to Editor, Dr. Erickson pointed out that t he citation anal ysis defined in the Ginther study w a s not relevan t to competitive scientists, the number of citations under consideration should be about 1,000, instead of being about 84, and t he number of citations should be normaliz ed to the career len gth. T he opini on was ex pressed that simi larly qualified scientists “ would be equall y successful in grant f unding, with no disparity for race and e thnicity ” [5]. D. K. Ginther et al . wrote a defensive response to these letters. They disagree wi th Voss about his explanatio n, because “ there is substantial evidence that affirmative action does not explain the results ” [6] . They found that “ blacks and whites were equal ly likely to r e ceive tenure at highe r educati on institutions that are research intensi ve ” , and “ a b ad match for research careers will have most likely been weeded out earlier ” . “ T here is a case to be made for positive selection of black scientists – that they are the best of the best – as opposed to being bad matches resul ting from affirmative action .” Also, they disagree with Erickson ab out the citation issue , because their data incl uded about 300 early-career individual s who had ~1,000 citations, being in the top 1% of the pool. Furthermo re, a recent evaluation o f the NIH K program [7] show ed that awardees published about 10 papers in the 5 years after the award and attracted about 15 0 citations per person. Furthermore, they di d not think that a g e-norma lizing citations would change their result s for early-career inv estig ators . Sherley com mented on the view fr om Taba k and Coll ins [2] , “ the limited p ublic discussion on the p ossible underlying f actors has focused on the NIH review process. Although this is an obvious place t o continue the inve stigation, the explanatio n ma y lie elsewher e ” [8] . “ Ba rriers at the ho m e institution ” were mentioned for “ minori ty investigator s pursuing primarily cancer health disparities research ”. For example, “ although NIH requires the writing of minority recruitment plans by its grantee institutions , it currently neither evaluates how nor even whether such plans are implemen ted .” Collins and Tabak did not agree w ith Sherley, “ the plans on a ll NRS A training grants are rigorously reviewed, and if they are deficient, the grants are not funded until corrective action is t aken on the part of the gra ntee. Awarded training grants that are subsequently submitted for renewal are reviewed for the recruitment plan’s results. If the plans are judged ineffec tive, this assess ment affects its li kelihood of being funded again ” [9]. Based on the above results and opinions, it is clear that the Ginther study [1] has a tremendous social influence and major policy implications but sev eral countervaili ng opinions r emain un reconciled. He re this issue is re-examined with a new approach and solid data, offerin g a pers pective f rom paired statistical analysi s on NIH f unding normali zed to individual’ s scientific productiv ity . It is the pairing and the normaliz ation components of the approach t hat allow deeper insight and more objective conclusio ns. In the next section, an apparent inconsis tency is commented on between the data and con clusions derived by Ginther et al. , and an alternative experimental design is proposed. I n the third section, an axiomatic approach is defined for quanti f ication of indiv idualized scientific producti vity . In t he fourth section, the experimental design is described al ong w ith key results. In the last section, relev ant issues a re disc ussed. 2. IS THE GINTH ER STUDY SELF-CONSISTENT? As a general p rinciple, an eq uitable distribution of research funding sh ould be proportional to each applicant’ s research cap abi lity measured by their scienti fic productivi ty. Scholarly publications are a widely-used produc tivity benchmark, and can be individual ly quantified by the h -index [ 10 ] or e quivalently number of citations. By this popul ar m etric, t he data i n t he Ginther s tudy [1] does not s uggest any significant un f airness in t he NIH revi ew process. Specifically , the av erage white appl icant had 78 citations , Page 3 while the average black applican t had 40 (p. 101 8 in [1]). Q uite proportionally , the average RO1 success rate was 30%- for white applicants and 15%+ for black (see Fig. 1 in [1]) . Th is citation-based proportionali ty an d the 1 0% disadvanta ge noted by G inther et al . see m i n contradiction, and mo tivated the authors to study the issue in more detail. One potential pitfall of the experimental design by Ginther et al . is a sub-op timal use of t he Probit m odel [1], which transforms a continuous sum of wei ghted variables to a binary outcome (funded or not in this case) via a Gaussian distribution to test t he association of race/ethnicity to RO1 success. The variabl e definitions, inner product (weighted sum), and the Gaussian form of the P robit model could be sub ject to deficiencies and m isma tches, especially for co mplicated problems. Most remarkably , the criterion for fairness i n the funding p rocess has not been well f ormulated in [1] . T her efore, it seems reasonable to revisit t his problem in a more effectiv e fashion. In the f ollow ing , paired t -tests will be used with rigorous matching criteria t o spot light any racial difference, and informativ e f eatur es especially funding success normalized by scienti f ic p roductivity w ill be ex tr acted to inv estig ate the fairness o f the NIH rev iew process. 3. HOW T O QUA N TIFY INDIVIDUAL SCIENTIFIC P RODUCT IVITY? While citation count and impact factor are popular measures of publicati on quality , there is no common agreement on ho w to quantify relativ e contributions among co-autho rs. T he number o f researchers, publications, and co-authors have all steadily increased over past de cades [ 11 ]. Consequently , the competition for acade mic resources has intensi f ie d, along with bu dget squeez es f rom the current f i nancial crisis. T o optimize the resource allocation , individualiz ed assessment of research results is being activel y studied [ 10 , 12 - 17 ] . H owever , cu rrent indi ces, suc h as the numbers of pap ers and citati ons, as well as the h -factor and its variants [ 10 , 14 ] have l imitations, especially their inabil ity to quantify co- authors’ credit shares ob j ecti vely [ 18 ] . R ecently , an ax iomatic system has been p roposed for quantification of co- authors’ credits, and the correspo nding estimation has been f ormulated [ 19 ] , This methodology allows the axiomatical ly fair m easu rement o f individuals’ publ ication records, av oiding sub jective assi gnment of co- authors’ credits usi ng the infla ted, fractional or ha rmonic methods . These findin gs can be i ncorporated into existing bibliometric indices f or enhanceme nt of their p redictive values [ 20 ], and has a potential to transform bibliometrics towards a rational framework, providi ng accurate and practical tools for scienti f ic management. A recent topic in bi bliometrics is the u se and ex tension of the h - index [ 10 , 14 ] f or measurement of t he productivity and impact of a researcher. W hi le it is increasingly used [ 21 - 25 ], the h -index is appro ximate by def inition [ 26 ], and subject to v arious biases [ 20 , 27 - 35 ]. A major obst ac le to si g ni ficant imp r ov ement o f the h -index and other popu lar indices of this ty pe has been the lac k o f assessmen t o f co- authors’ individual contributions. It is well recognized that the quantification of indivi dual co - authors’ credits in a publication is extremely important [ 12 , 13 , 15 - 17 ] . Current perception of a researc her ’s qualification relies heav ily on either i nflated or fr actional counting methods [ 36 ]; while t he f orme r method g ives the full credit to any co -author , the latter method di stributes an equally divided credit to each co -author . Neither of the se methods is ideal because the o r der o r rank o f co-authors, and t he corr espondin g au thorship, are no t used t hat indi cate the relative contributions of co- authors. Generally speaking, the further down the list of co -authors for a publication, the less credit he or she receiv es ; the f irst and corresponding authors are consi dered the most prominent . Page 4 The harmonic counting method was proposed [ 36 ] in order to avoi d the equal-share bias of the fr action al counting method (a less sophisticated variant was also su gg ested [ 17 ]). W hil e the har monic counting method does permit equal rankings for subsets of co-author s, let us assu me that the order of co- authors’ names is consis tent with their c redit ranking, and that there a re n co-auth ors on a publ ication w hose shares are presen t ed as a vector 12 ( , , , ) n x x x x  ( 1 in  ). Then, t he k- th author contributes 1/ k as much as the first author . Realistically , there are many possibl e ratios between the k -th and the first author ’ s credits, w hich m ay be equal or may be rather sm all (e.g. cases of data sharin g or technical assistance). Hence , the harmonic method has ne ver been used in practice. There are critical and immediate needs f or rigorous quantification of co - authors’ credits . The Higher Education Funding Council f or England (HEFCE) recently proposed the peer - review system “ Re search Excellence Framework (REF) ” [ 37 ] that wil l utilize citation analyses. Nevertheless, HEFCE has admitted that bibliometrics is not " sufficiently robust " for assessment of research quality . Thus, it could be prone to misconduct if bibliometric measures are directly used f or funding and tenure decisions. For example, a popular Chi nese w eb forum “ New Thread s ” [ 38 ] discussed several cases of ar tificially inflated numbers o f publications, co-authors, and even h -index es . In the USA, the National Institutes of Health recently adopted en hanced review cri teria [ 39 ] , w ith mandatory quantification of an investigator ’s qualification on a 9-point scale (revised from the initially planned 7-point scale) ; how ever , the scoring has been largely subjectiv e. Assume that each publi cation has n co-au thors in groups ( ) where co-au thors in the i -th group have the sa m e cre dit ( ). We pos t ulate the followi ng three axioms: A xiom 1: ; A xiom 2: ; A xiom 3: is uni formly distributed in the dom ain defined by Axio ms 1 and 2. While the f irst two axioms are sel f-evident, the third asse rts t hat all the cases permitted by Axioms 1 and 2 are equally possible b y the maximum entropy principle [ 40 ]. Therefore, t he fairest estimation of co- authors’ credits must be the expecta tion o f all possi ble credi t vectors. In other words, the k- th co- author’s credit must be the corresponding elemental mean, which has a closed form expression [ 19 ], which is referred to as the a -index for its axiomatic foundation. Naturally , th r ee ind ividualized scientific producti vity measures can be de fined. First, the prod uctivity measure in t e rms of j ournal reputation, or the Pr -index, is the sum o f the jou rnal impact f acto rs ( IF) of one’s papers weighted by his/her a -indices res pectively. Second, the productivity measure i n terms of peers’ citations, or the Pc -index, is the numbers of citations to his/her papers weighted by a -indices respectively . W hile the Pr -index is useful for immediate productiv ity measure ment, the Pc -index is retrospective and generally more r elevant. Finally , the P c*IF index the sum of the numbers of citations after be ing individually weighted by both the a -ind ex and jou r nal impact f actor. W hen papers are cited, the Pc*IF index credits hi g h-impact j ournal paper s more than low-impact c ounterparts, as highe r-impact papers generall y carry tighter relev ance or o ffer stronger suppo rt to a citing pape r. m nm  c i ( , , , ) 12 x x x x x m i  1 im  0 12 x x x m     1 1 1 2 2 c x c x c x mm    x Page 5 4. DOES RA CIAL BIAS EXIST IN T HE NIH REVIEW PR OCESS? 4.1. Human Subjects This study targeted t he top 92 American medi cal school s ranked in the 2011 US News and W orld Rep ort , from which the 31 odd -number-ranked schools were selected for paired analysis (schools were e xcluded if they did not provi de online faculty photos or did not allow 1:2 pairing of black versus w hite faculty members). Data were gathered from Sep tember 1 to 5, 2011 on black and w hite f aculty m embers in departments of inte rnal medicine, sur g ery, and basic sciences in the 31 selected schools . W hite and black/African A merican faculty members w ere confirmed by thei r photos, names, and resumes as needed , and depar tment heads/chairs w ere excluded. These scho ols were ca t egoriz ed i nto t h ree t iers according to their ran king: 1s t-31st as t he first tie r, 33rd-61st as t he second t ier, and 63rd -91st as the third tier. After 130 blac k f ac ulty members were foun d from these schools, 40 black f aculty members were randomly selected. With the pairin g c riteria including the same gender, de gree, title, specialty and university , t he selected 40 black faculty members were 1:2 paired w ith white peers, yieldi ng 120 samples as our first pool. Among the 130 blac k samples in the initial li st, 14 faculty members w ere funded by NIH during the period from 2008 t o 2011. Two of 14 black samples were excluded because of failure in matching with a white faculty. Furthermore, an additional black faculty member was excl uded because he only published at conference without any Science Citation Index (SCI) record in this period [ 41 ]. Consequentl y, 11 funded black f acul ty members were kept. Among them, 10 were from the f i rst t ier, and 1 f rom the second tier. These 11 funded blac k f aculty members were 1:1 paired w ith whi te samples who both met t he pairing criteria and were funded by NIH in the same perio d. Consequentl y, there w ere 11 pa irs o f black an d w h ite investigators, w hich is our second pool. 4.2. Data Analysis Using the Web of Know ledge [ 41 ] , datasets were systematically collected f or the two pools of facu lty members. Each dataset corresponded to a single black-white combination, and included bibliographic information, such as co-authors, assignment of the corresponding author(s), journal impact factors, and citations 2008-2011. The journal impac t factors w ere obtained from Journal Citation Reports [ 42 ] . The a -index values were computed using the formula derived by W ang and Yan g [ 19 ]. In computing a -index value s, t he first author(s) and the corresponding author(s) were treated with equal weights in this context. For the NIH-funded samples, indiv idual numbers of funded proposal s and individu al funding totals were found vi a the NIH Report er system [ 43 ] . Our features of interest included the number of journal papers, number of citations, Pr -index, Pc -index , and P c*IF -index . In addition, for the second pool samples addi tional f eatures were numbers of NIH funded proposals and NIH fundin g totals per person and pe r racial group , respectively. The paired t-tests were performed using SPSS 13.0 on the datasets from the first and second pools . In the first pool , the average dat a of two white pro fessors were paired to individual data o f the co r respondi ng black pro fessor. The tests were speci f ically performed by pro fessional r ank and scho ol repu tation, gender an d integrated for racial groups. Page 6 4.3. Key Res ults The scientific produc tivity w as evaluated using the Pr -index, Pc -index , and Pc*IF . Statistical significance levels are indicated by “* ” for p <0.05 and “**” for p <0.01 . T able 1 su gg ests that higher sci entific produc tivity was positiv ely correlated w ith more seni or profess ional titles or more prestigious institutional t iers. Furthermo re, the analysis shows male investigators were statistically more productive than the female colleagues, and black faculty members statisticall y less productive than white colleagues. The distribution of pro fessional titles (Full, Associate, and As sistant Professor) for black f aculty members was 3:12:25, indicating an imbalance in the higher ranks. Despite that more than a half of the black samples were from first tier institutions, 14 were assistant professors. Thus, the numbers of black a ssociate and full professors were insufficien t for us to devise t itle- speci fic conclusions w ith statistical signi ficance. T able 2 focuses on the scientific productiv ities of the NIH f unded black and white investigator s, and indicates similar racial differences in scientific productiv ity . Although statistical significance cannot be established per p rofessional title due to the li mited numbers of sampl es, t he dif f erences between the ra cial groups are significant i n te rms o f t he numb er o f citations and the Pc -ind ex. In the following anal ysis , these scientific productivity measures will serve as t he base to evaluate the fairness of the NIH funding process. Note that the racial/ethnic dif f erences in Pr and Pc (T ables 1 and 2) are consistent with the citation analysi s performed i n [1]. In T ables 3 and 4, the funding support and the number of funded pro jects for each racial grou p w ere normalized by Pr , Pc and Pc*IF respe ctively . In addition t o the racial differen ce in the RO1 success rates [1], it can be seen i n T ables 3 and 4 tha t the f undi ng total an d the number o f funded projects for black NIH investigators w ere only 46% and 62% of that for w h ites , respec t iv ely . However , when these f undin g totals and numbers of funded proj ec ts w ere no rmalized by Pr , the ratios between black and w h ite faculty members were narrowed. Furthermore , the normalization by the ci tation-oriented indices Pc an d Pc*IF indicates that blac k faculty members had more favorable ratios fro m 1.06 to 2.00 . Page 7 Race Number of Samples Mean Mean of Papers Number of Citations Pr -index Pc -index Pc*IF-index Full Black 3 16.33±17.24 120.67±144.36 17.62±23.21 33.24±50.06 130.51±202.80 Professor White 6 17.67±22.87 197.83±279.04 17.49±19.77 20.96±26.88 260.35±326.53 Associate Black 12 5.83±5.75 30.00±37.10 4.73±5.25 4.69±5.35 31.32±42.73 Professor White 24 9.08±8.63 52.25±55.76 5.38±4.55 7.78±6.04 41.23±58.22 Assistant Black 25 2.44±3.11** 8.88±20.35* 1.71±2.17** 0.86±1.29* 2.87±5.49* Professor White 50 5.18±4.86 31.94±52.94 6.05±6.42 7.05±11.23 48.42±107.01 First Tier Black 21 5.19±8.18** 27.62±63.63* 5.29±9.92* 6.09±19.63 29.13±82.78 (Groups 1- 21) White 42 10.02±10.66 70.31±118.28 9.22±9.38 11.07±14.88 87.12±168.07 Second Tier Black 8 6.00±6.28 36.50±45.26 3.41±3.36 4.91±6.08 24.14±29.35 (Groups 22-29) White 16 5.69±5.32 26.44±26.85 6.20±5.51 6.71±5.77 37.82±51.48 Third Tier Black 11 2.09±1.81 6.55±8.66 1.26±1.42 0.94±1.38 3.12±6.82 (Groups 30-40) White 22 3.23±2.79 30.09±53.54 2.28±2.33 4.21±6.10 32.22±64.83 Male Black 22 6.14±7.91* 36.55±65.60 4.72±9.17** 6.60±19.27 32.58±81.54* White 44 9.68±10.42 66.25±111.14 8.79±8.82 9.93±11.21 75.90±135.35 Female Black 18 2.50±4.16 7.78±11.79 2.69±4.71 1.79±2.93 6.81±11.68 White 36 4.36±4.50 31.19±59.12 4.16±5.60 6.33±12.44 45.37±123.49 Total Black 40 4.50±6.68** 23.60±50.87* 3.81±7.49** 4.44±14.48 20.98±61.71* White 80 7.29±8.63 50.48±92.12 6.71±7.81 8.31±11.77 62.16±129.42 Ratio 0.5 0.62 0.47 0.57 0.53 0.34 T able 1: Scientific producti vity measures for black and white faculty members in t he first pool. Race Number of Samples Mean Number of Papers Number of Citations Pr -index Pc -index Pc*IF-index Black 11 10.45±9.02 88.64±98.30* 11.13±12.47 14.96±24.11* 90.43±124.94 White 11 18.64±14.18 203.73±189.02 18.03±13.24 34.39±43.82 318.42±474.53 Ratio 1 0.56 0.44 0.62 0.44 0.28 Table 2: Scientific pro ductivity measures for b lack and white faculty memb ers in the second po ol. Table 3: Ratios between the tot al funding amount and the accumulated scientific productivity for racial groups (not individuals) in the s econd pool. Race Number of Samples Funding Total Funding Total Normalized by Pr -index Funding Total Normalized by Pc -index Funding Total Normalized by Pc*IF-index Black 11 20140082 164565.69 122423.76 20247.54 White 11 43796537 220860.92 115781.91 12503.74 Ratio 1 0.46 0.75 1.06 1.62 Page 8 Race Number of Samples Number of Projects Number of Projects Normalized by Pr -index Number of Projects Normalized by Pc -index Number of Projects Normalized by Pc*IF-index Black 11 22 0.180 0.134 0.022 White 11 37 0.187 0.098 0.011 Ratio 1 0.59 0.96 1.37 2.0 Table 4: Ratios betwe en t he to tal number of funded projects an d the accumul ated s cientific product ivity f or racial groups (not ind ividuals) in the second po ol. 5. DISCUSS ION AND CONC LUSION There are apparent differences in re search performance by major r aci al groups based on indiv idual scientific produc tivity measures. T hese findings are consistent with prev ious reports [1]. The application of the new scientific produ ctivity indi ces t o the racial groups (T ables 1 and 2) clarifies the source of discrepant funding success . W hen the t otal grant amoun ts and the number of funded projects were racial-group-wise normaliz ed by these indices, the NIH review process does not appear biased agains t black f acul ty members (T ables 3 and 4). Specifical ly , the f undin g total and the number of f unded projects for black NIH investigato rs were respectivel y onl y 46% and 62% of that for white peers. However , when these funding totals and the number of funded projects were normaliz ed by Pr , the ratios betw een black and white f aculty members neared parity . Furthermore, the normalization by the citation-oriented indices Pc and Pc*IF indicates that black resea rchers are not in a disadv antageous positi on. The axiomaticall y derived a- , Pr - , Pc -, and Pc*IF- indices individual ize credits (journal impact factor , number of citations, or both) for coauthored papers or other forms of joint teamwork. These metrics apportion an i ntegrated cont ribution most e q uitably among resea rchers so that credit c an be quantitativel y shared f or team science activity . All figures of merit includi ng ax iomatically derived ones have limi tations but assessment of scientific pro ductivity and research potential should be done to be commensurate with individual contributions . Original ity , novelty , healthcare impact, and peers’ perception are all critical facets of the assess ment. The axiomatic approach is advan tageous due to rigor and objectivity , should be positively correlated to the other quantitative and qualitative criteria, and could be helpful in the NIH f undin g process to q uantify achiev ements, detect disparity , and facilitate management . In particular , such tools could aid streamli ning and monito r in g of peer-review and research execution. The key results achieved statistical sig ni ficance , w hen subjec ted to pa ired analysis capable o f sensing dif ferences with adequate specificity and sensitiv ity . There is potential for the axiomatic approach to produce more compreh ensive results with expansion o f t he sample siz e . The databases construction used i n this study too k our 10 students’ e ffo rts ov er about three mon t hs, and yet cannot be compared with that used in t he Ginther study in terms of sample size (The Ginther study wa s based on a much larger sample siz e , “ this sample included 83,18 8 observations with non-missing data for the explan atory variables ” [1]). On the other hand, if there w ere detailed in formation on education al bac kground, training, prior awards, and related variabl es, pairing of black and whi te investigators could become impossible in many cases. In this study , the cr i tical abstraction across various groups has been axiomatically -formulated scientific productiv ity and accordingly-defined funding normaliz ation . This perspective all ows us to ev aluate the fairness of the NIH review process in a more straightforward way . The limitations of the current study are multiple, and have compromised the results to dif ferent degrees. The research discipli nes, specific institutes , othe r grant m echanisms (e.g., P and K awards) were not Page 9 separately considered. T he prior training (T and K aw ards), longitudin al trends , and rev iew p rocess changes were not analyzed . W hen the samples were selected, the unavai lability of some faculty photos was a dif f iculty . Since the number of whi te faculty members is large , it was hoped to use more w hite samples for a better representation. However , the pairing criteria prevented us from incl uding white faculty members beyond the 1:2 and 1:1 ratios for the f irst and second pools, respectively . The existin g online sea rching system s do not sup port the computation o f the axiomatic indices. The tedi ous data entry and analysis t asks are error-prone. Cross validation steps were performed to produce data up to a high standard. Ideally , an aut omated ex clusive study using the ax iomatic app roach shoul d be performed to generate the hi ghest possi ble statistical con fidence. In this re g ard, the Gint her study is a model. Axiomatical ly-oriented bibli ometrics employ s value theory to address the basic question of how , why and what v alue is ascribed to an indi vidual's scientific w ork . Cul tural, soci ological, g eographi cal, psychological, economical, physical, computational, and other factors influence the resul ts. In the 19th century , Adam Smith asserted that the amount of labor put into a physical product determine d its exchange value. T he concept was refined by others, including Karl Marx, John Keyne s, and the Chicago school of econo m ics, but the valuation of an intell ectual product is m u ch more chal lenging. Alt hough extensive studies have been done on this topic, including citation analysi s, there has been no reliable means t o value individual credits in t eamw ork or joint publications. An axiomatic t heory for individualized quantification of scien tific productivity [ 19 ] introduced to add r ess this need, was used in this racial dispa rity study . In the f uture, major search en gines such as Web of Science and Google Scholar may implement the Pr - , Pc - and Pc*IF -indices to augment indiv idual productiv ity assessments. Although the NIH review process endeavors to be racially f air , i t is not perfect in a ll aspects. How to evaluate and optimize the NIH funding process has been a hot topic [ 44 ]. The N IH Grant Productivi ty Metrics and Peer Review Scores Online Resource [ 45 , 46 ] s timulates hypotheses t hat can be tested using the axiomatic indices. For example, wil l new in vestigators be more influential than senior researchers? W ill large grant mechani sms such as U01 and P41 be more productive than R01 and R21? Will renewed projects be more cost-effectiv e than initially funded projects? Although any bibliometric measures are subject to inter-specialty fluctuations, so me of co mm onl y interested problems c an be studied using the ax iomatic indi ces with the same indivi dual or team as its own control. In conclusi on, the NIH grant r acial disparity study of Ginther et al. [1] was augmented by a pairin g-based axiomatical ly-individualized productivi ty and nor malized f unding success measurement tr ial. Although there are racial di fferences in R01 grant succe ss rates [1], nor malized f igures o f merit for fundin g success explain the discrepancy . T he suggested “ leverage points for policy intervention ” [1] a re i n q uestio n an d require deeper and more thorough investigations, given t he important social complications of this sensitive issue. Further adjustments in policies to remove racial disparity should be made more systematically f or equal opportunity , rath er than b eing limited to the NIH rev ie w process. Page 10 A cknowledgment The authors are grateful f or diligent data entry and analysis work performed by our research assistants especially W.Y . Wang, S.H. T ong, Y .Y . Gao, and G.C. Liu. Page 11 References 1. Ginther , D.K., et al., Race, Ethnicity , a nd NIH Research A wards. Science, 201 1. 333(6045): p. 10 15 -1019. 2. T abak, L.A. and F .S. Collins, Weaving a Richer T apestr y in Biomedical Sci ence. Science, 201 1. 333(6045): p. 940-941. 3. Kaiser , J., NIH Uncovers Racial Disparity in Gr ant Awards. Scienc e, 201 1. 333( 6045): p. 925 - 926. 4. V oss , J.L., Race Disparity in Gr ants: Empirical Solutio ns Vi tal. Science, 2 01 1. 334(605 8): p. 899 -899. 5. Erickson, H.P ., Race Disparity in Grants: Check the Citat ions. Scienc e, 201 1. 3 34(6058): p. 899 -899. 6. Ginther , D.K., et al., Race Disparity in Grants: E mpirical Solutions V ital Response. Scie nce, 201 1. 334(6058): p. 899- U154. 7. http://grants.nih.gov/tra ining/K_Awards_Evaluation_FinalR eport_201 10 901.pdf. 8. Sh erley , J.L., Race Dispari ty in Grants: Oversight a t Home. Science, 201 1. 334(6 058). 9. Collins, F .S. and L.A. T abak , Race Disparity in Grants: Oversight at Home Respon se. Science, 201 1. 334(6058). 10. Hirsch, J.E., A n index to qu antify an individual's sc ientific res earch output. Proc N atl Acad Sci U S A, 2005 . 102(46): p. 16569- 72. 1 1. Greene, M., The dem ise of the lone author. Nature, 2007. 450(7173): p. 1 165. 12. Foulkes, W . and N. N eylon, Redefining authorship. Rel ative contributi on should be given after each author's name. BMJ, 1996. 31 2(7043): p. 1423. 13. Campbell, P ., Policy on pap ers' contributors. Nature, 1 999. 399(6735): p. 393. 14. Hirsch, J.E., D oes the H index have predictive power? Proc Natl Acad Sci U S A, 2007. 104(49): p. 19193 - 8. 15. Anonymous, Who is accountable? Nature, 2 007. 450(7166): p. 1 . 16. Ball, P ., A longer paper gathers more citations. Nature , 2008. 455(721 1): p. 274 - 5. 17. Zhang, C.T ., A proposal for calculatin g weighted citations based on author rank. EMBO Rep, 2009. 10(5) : p. 416 - 7. 18. Williamson, J.R., My h -index turns 40: my midl ife crisis of impact. ACS Chem Bi ol, 2009. 4(5): p. 31 1 - 3. 19. Wang, G. and J.S. Y ang, Axiomatic quant ification of co -authors' relative contribu tions. arXiv .org, 2010: p. arXiv:1003.3362v1 [sta t.AP]. 20. Bornmann, L. and H.D. Daniel, The state of h i ndex research. Is the h index the ideal way to measure research performance ? EMBO Rep, 2009. 10(1): p. 2- 6. 21. Ball, P ., Index aims for fair rank ing of scientists. Nature , 2005. 436(7053): p. 900. 22 . Ball, P ., Achievement index climbs the ranks. Nat ure, 2007. 448(7155): p. 737. 23. Kinney , A.L., National scientific fac ilities and their scienc e impact on nonbiomedica l research. Proc Natl Acad Sci U S A, 2007. 104(46): p. 17943- 7. 24. Pilc, A., The use of citation indicators to identify and su pport high -quality research in Poland. Arch I mmunol Ther Exp (Warsz), 2008. 56(6): p. 3 81- 4. 25. Sebire, N.J., H-index and impact factors: assessing the clinical impact of r esearchers and specialist journals. Ultrasound Obstet Gy necol, 2008. 32(7): p. 84 3- 5. 26. Dodson, M.V ., Citation analys is: Maintenance of h- index and use of e-index. Biochem Biophys Res Commun, 2009. 387(4): p. 625- 6. 27. Lehmann, S., A.D. Jackson, and B.E. Lautrup, Measures for measures . N ature, 2006. 44 4(7122): p. 1003 - 4. 28. Jeang, K.T ., Impact fac tor , H index, peer compariso ns, and Retrovirology: is it tim e to individualize citation metrics? Retrovirology , 2007. 4: p. 4 2. 29. Kelly , C.D. and M.D. Jennions, H -index: age and sex m ake it unreliable. N ature, 2007. 449(7161): p. 403. 30. Wendl, M.C., H-index: how ever ranked, citations nee d context. Nature, 2007. 449(7161): p. 403. Page 12 31. Engqvist, L. and J.G. Fr ommen, The h -index and self-c itations. T rends Eco l Evol, 2008. 23(5): p. 250 - 2. 32. Mishra, D.C., Citati ons: rankings weigh agai nst developing nations. Nature, 2008. 451(7176): p. 244. 33. Radicchi, F ., S. Fortunato, and C . Castellano, Universality of citation distributions: toward an objective measure of scientific i mpact. Proc Natl Acad Sci U S A, 2008. 105(45): p. 17268 -72. 34. T odd, P .A. and R.J. Ladle, Citations : poor practices by authors reduce their value. Nature, 2008. 451(7176): p. 244. 35. Baldock, C., R. Ma, a nd C.G. Orton, Point/cou nterpoint. The h index is the best measure of a scient ist's research productivity . Med Phys, 20 09. 36(4): p. 1043 - 5. 36. Hagen, N.T ., Harmon ic allocation of authorship c redit: source -level correction of bibliometric bias assures accurate publication and citation analysis. PL oS One, 2008. 3(12): p. e40 21. 37. http://www .nature.com /news/2009/090 923/full/news.2009.933.h tml. 38. http://www .xys .org/new .html. 39. http://grants.nih.gov/gr ants/guide/notice- files/NOT - OD - 09 -02 4.html. 40. Jaynes, E.T ., On the R ationale of Maximum -Entropy M ethods. Proceedings of the Ieee, 19 82. 70(9): p. 939 -952. 41. http://sub3. webofknow ledge.com . 42. http://thomsonreuters.c om/products_services/sc ience/science_products/a - z/journal_citation_reports . 43. http://projectreporter.nih.gov/reporter.cfm. 44. https://loop.nigms.ni h.gov/index.php/category/p eer-review . 45. https://loop.nigms.ni h.gov/index.php/201 1/06/02/produ ctivity- metrics- and - peer -review-s cores. 46. https://loop.nigms.ni h.gov/index.php/201 1/06/10/produ ctivity- metrics- and - peer -review-s cores-continued.