End-to-end research evaluation needs to separate out the bibliometric part of the chain from the econometric part. We first introduce the role of size-dependent and size-independent indicators in the bibliometric part of the evaluation chain. We show that performance can then be evaluated at various levels, namely a zeroth-order, a first-order or even a second-order. To complete the evaluation chain, we take up the econometric part where efficiency of the research production process is represented in terms of output and outcome productivities. Both size-dependent and size-independent terms play a crucial role to combine quantity and quality (impact) in a meaningful way. Output or outcome at the bibliometric level can be measured using zeroth, first or second-order composite indicators, and the productivity terms follow accordingly using the input to output or outcome factors.
An evocative analogy for understanding the relationship of size-dependent to size-independent factors in all measurement is Archimedes' discovery of the concept of density. The density ρ is a size-independent term that allows the weight W to be computed from the volume V, which is the primary size-dependent term. Note that now, W combines both size-dependent and sizeindependent terms into a meaningful composite secondary indicator. The bibliometric parallel for this are P, the number of publications and C the number of citations in a portfolio of publications. Thus, if P is taken as the primary bibliometric indicator of size, then C becomes a secondary and composite bibliometric indicator of performance. Impact, which is represented by i = C/P, is a natural candidate for a size-independent proxy for the quality of the portfolio.
Of course at this stage, we assume that all publications are in the same discipline and from a coeval window so that normalization is not an issue. Normalization is only an additional detail that can be rationally worked out (Ruiz-Castillo & Waltman 2015).
If C is thought of as a first-order indicator of performance, then it is possible to bring in the idea of an higher-order energy-like term X = iC = i 2 P, as another indicator of bibliometric performance. Thus, C combines impact i and output P by weighting each publication with its citation impact. The I3 indicator (Leydesdorff & Bornmann, 2011) combines normalized impact and output and is therefore a first-order indicator of performance. The exergy indicator of Prathap (2011) is a second-order indicator of performance. P, standing alone, is then a zeroth-order indicator of performance. Thus all three, P, C and X are valid measures of output or outcome depending on the extent to which one wants to give weightage to the quality proxy, in this case, the impact i.
Let us now come to the econometric part of the chain. We need a meaningful measure of input as this is crucial to the calculation of the research efficiency or productivity of any researchintensive unit. In 2014, SIR introduced a new feature that makes end-to-end evaluation from input to output possible. This is called the scientific talent pool (STP), which gives the number of authors from an institution who have participated in the total publication output of that institution during that particular period of time. Savithri and Prathap (2015) used this indicator as a reasonable proxy of the input at the beginning of the chain that performs scientific research activity.
To the best of this author’s knowledge, Hendrix (2008) was one of the earliest to evaluate institutional-level performance of research units by intelligently classifying and clustering various bibliometric indicators using Principal Component Analysis (PCA). The variables clustered neatly into three distinct groups: the first cluster comprise size-dependent input and output terms, namely the total number of faculty (input), total number of papers (output), and total number of citations (outcome). The second factor comprised size-independent terms that reflect the impact of a researcher, average number of citations per article, etc. and can be interpreted as a quality or excellence dimension. The third group, also influenced heavily by size-independent terms, describes research productivity and impact at the individual level, like the number of papers and number of citations per faculty member. Savithri and Prathap (2015) used the PCA approach to show that with only five variables, two components suffice to account for most of the common variance. These are the size-dependent quantity indicators and the size-independent quality and productivity indicators, which are clearly orthogonal to the former. This allowed representation and visualization of the primary and secondary data as two-dimensional maps. Thus for an end-to-end evaluation, sizedependent and size-independent indicators play a very critical role.
In this paper, we represent the indicators needed for the complete end-to-end chain as shown in Table 1. Using this we rework the simple example in Abramo and Angelo (2016). Table 2 takes the case of two universities of the same size (say 100 Full time Equivalent Researchers or FTERs), resources and research fields. Unit A publishes 100 articles earning 1000 citations (i.e. impact of 10 citations per article). Unit B publishes 200 articles, and gathers a total of 1500 citations (i.e. average impact of 7.5 citations per article). The last column of Table 2 shows the efficiency or effectiveness advantage of B over A using the Mean Normalized Citation Score (MNCS) approach. Since performance is a multi-dimensional construct, we have different results -A is better than B on quality alone, but on output or outcome productivities, depending on the choice of order of indicator, the advantages change. The exercise can be repeated using the Highly Cited Articles (HCA) approach. Unit A has 10 HCAs while Unit B
This content is AI-processed based on open access ArXiv data.
