Probing Multivariate Indicators for Academic Evaluation

Helen F. Xue1, Loet Leydesdorff2, Fred Y. Ye 3*

1 Library, Zhejiang University, Hangzhou 310027, CHINA 2 Amsterdam School of Communication Research (ASCoR), University of Amsterdam,
PO Box 15793, 1001 NG Amsterdam, The Netherlands 3 School of Information Management, Nanjing University, Nanjing 210023, CHINA

Abstract: We combine the Integrated Impact Indicator (I3) and the h-index into the I3-type framework and introduce the publication vector X = (X1, X2, X3) and the citation vector Y = (Y1, Y2, Y3) , the publication score I3X=X1+X2+X3 and the citation score I3Y=Y1+Y2+Y3, and alternative indicators based on percentile classes generated by the h-index. These multivariate indicators can be used for academic evaluation. The empirical studies show that the h-core distribution is suitable to evaluate scholars, the X1 and Y1 are applied to measure core impact power of universities, and I3X and I3Y are alternatives of journal impact factor (JIF). The multivariate indicators provide a multidimensional view of academic evaluation with using the advantages of both the h-index and I3.

Keywords: I3; h-index; publication vector; citation vector; publication score; citation score; multivariate indicator; academic evaluation

• Corresponding author: Fred Y. Ye, Email: yye@nju.edu.cn 2

1. Introduction Academic evaluation has continued to be an issue in the academic world, as it is difficult to select and set universal evaluating principles in various complicated situations. However, publications and citations remain the main focuses of academic evaluation, particularly for fundamental research. Citations cannot directly be compared with publications and thus one needs a model or at least a formula. A model can be improved and thus the measurement be refined. Since all models also generate error, the quality of a model depends on the quality of the arguments used for constructing the model.
   Since Garfield introduced the journal impact factor (JIF) and set up citation analysis (Garfield, 1955, 1979), these scientometric indicators have been applied to academic evaluations. Hirsch (2005) proposed the h-index, which was rapidly accepted by the scientific community. This promoted the development of quantitative academic indicators. However, both JIF and h- index have their advantages and disadvantages. JIF is basically designed for journals and the h- index for the evaluation of individual scholars.
   After developing a set of criteria for an indicator in Leydesdorff et al. (2011), these authors proposed the Integrated Impact Indicator I3 (Leydesdorff & Bornmann 2011). I3is based on (i) transformation of the citation distribution into a distribution of quantiles and (ii) integration (instead of averaging) of the quantile values. (Quantiles are the continuous equivalent of percentiles.) The use of percentiles was recently recommended in the Leiden Manifesto (“Ten principles to guide research evaluation”; Hicks et al., 2015), because average citation rates are heavily dependent on the few highly cited papers in a publication set and bibliometric distributions are very skewed. I3 combines citation impact and publication output into a single number – similar to the h-index.
   3 The quantile values which are conveniently normalized between zero and hundred provide the weights for the papers, as follows:

    C i i i X X f i I 1 ) ( ) (3

(1)

where Xi indicates the percentile ranks and f(Xi) denotes the frequencies of the ranks with i=[1,C] as the percentile rank classes, which means that the measures Xi are divided into C classes each with a scoring function f(Xi) or weight (wi). One can also re-write Eq. (1) as follows:

1 ; ) ( 3     i i i i i w X w i I

(2)

As an alternative to quantiles, the h value of a document set can be used to provide a rank class structure. This combines the advantages of I3 and h into a single framework (Rousseau & Ye, 2012; Ye & Leydesdorff, 2014), which can be applied to academic evaluations based on publications and citations at both group and individual levels. In this study, we elaborate this methodology which was previously applied to journals (Ye et al., 2017), to universities as well as individual scholars.

2. Methodology In many cases, single numbers are used as indicators in academic evaluations. However, a single number can only reflect one side of the overall information and can therefore be expected 4 to have limitations and disadvantages. Possible solutions are multivariate indicators which reflect the multidimensional information. The h-based I3-type multivariate indicators provided a framework of such an elaborate methodology (Ye & Leydesdorff, 2014; Ye et al., 2017).

3.1 Methods Let us assume that t

…(Full text truncated)…