With the passage of more time from the original date of publication, the measure of the impact of scientific works using subsequent citation counts becomes more accurate. However the measurement of individual and organizational research productivity should ideally refer to a period with closing date just prior to the evaluation exercise. Therefore it is necessary to compromise between accuracy and timeliness. This work attempts to provide an order of magnitude for the error in measurement that occurs with decreasing the time lapse between date of publication and citation count. The analysis is conducted by scientific discipline on the basis of publications indexed in the Thomson Reuters Italian National Citation Report.
Deep Dive into Assessing the varying level of impact measurement accuracy as a function of the citation window length.
With the passage of more time from the original date of publication, the measure of the impact of scientific works using subsequent citation counts becomes more accurate. However the measurement of individual and organizational research productivity should ideally refer to a period with closing date just prior to the evaluation exercise. Therefore it is necessary to compromise between accuracy and timeliness. This work attempts to provide an order of magnitude for the error in measurement that occurs with decreasing the time lapse between date of publication and citation count. The analysis is conducted by scientific discipline on the basis of publications indexed in the Thomson Reuters Italian National Citation Report.
The use of national exercises to evaluate research systems is becoming ever more diffuse. One of the major objectives is to support efficient allocation of public resources to the various actors in the national systems. Traditionally the assessment exercises relied on peer review approaches, but advances in bibliometric techniques have led to many governments adopting bibliometric indicators to inform or even entirely substitute peer review, at least for the hard sciences. The penetration of bibliometrics can be appreciated by examining the typologies of three assessment frameworks: the Research Excellence Framework (REF) in the UK, the Quinquennial Research Evaluation (VQR) in Italy, and the Excellence in Research for Australia initiative (ERA). For the ERA, preparations for submissions began in June 2010. For the VQR, detailed guidance on submissions and assessment criteria is expected in 2011. For the United Kingdom REF, guidelines will be published during 2011, with institutions invited to make submissions during 2013 and actual assessment taking place in 2014. The REF is a typical example of a so called "informed peer-review" exercise, where the assessment outcomes will be a product of expert review informed by citation information and other quantitative indicators. It will substitute the previous Research Assessment Exercise series which were pure peer-review. The Italian VQR, substituting the previous pure peer-review Triennial Evaluation Exercise (VTR), can be considered a hybrid: a varying mix of pure peer-review, informed peer-review and the bibliometric approach. To prepare judgments of research output quality, the panels of experts appointed in each of fourteen disciplines, can choose one or both of two methodologies for evaluating any particular output: i) citation analysis; and/or ii) peer-review by external experts, selected by a collegial decision of the panel. The Australian ERA assessment in the hard sciences is conducted through a pure bibliometric approach. Single research outputs are evaluated by a citation index referring to world and Australian benchmarks. Because the entire research staff of the institutions must submit their full research product, indicators of research volume are also used to evaluate overall research performance.
Studies have demonstrated that there is indeed a positive relationship between citations of a work and the opinions of experts concerning its quality (Oppenheim, 1997;Rinia et al., 1998;Aksnes and Taxt, 2004;Reale et al., 2006;Franceschet and Costantini, 2011), however there are numerous differences between peer review and bibliometrics, including in the limitations of the two approaches.
Peer review presents a series of well documented and much discussed limitations regarding each of its three fundamental steps: i) the choice of products for submission to evaluation; ii) the choice of experts entrusted with evaluation of the products; iii) the inherent subjectivity in the judgments given by the reviewer, as offered for each product (Moxham and Anderson, 1992;Horrobin, 1990, Bornmann 2008). However bibliometrics also has its own limitations. The most notable is the fact that it can only be applied to disciplines where publication in journals is considered a reliable proxy of research output, meaning only the hard sciences (Moed, 2005). For the hard sciences, Abramo and D’Angelo (2011) have compared the results of the Italian VTR with those from a bibliometric simulation and have shown that bibliometric approach is greatly preferable to peer review for accuracy, robustness and functionality of measurement, and for the costs and times involved. However the 2011 study by Abramo and D’Angelo did not deal with the critical concern of the time that elapses between date of publication and the date of actually counting the citations, which is necessary to obtain citation counts that can give an accurate measure of the true publication impact. In theory the peer-review approach would permit an evaluation of quality immediately on release of the publication, but with bibliometrics the citations of a work can only be a good proxy of true impact when there has been a sufficient lapse from the date of publication. A minimum “citation window” is necessary. The potential problem is that whatever the intentions for the evaluation exercise (selective funding, informing research policies and management decisions, reducing information asymmetry between suppliers of knowledge and users), it is highly desirable that the evaluation results be available in close reference to the period being evaluated. This factor could affect the applicability of bibliometric methods. We note that in spite of the questions raised, the time necessary to implement peer-review exercises is longer (two years or more for the entirety of steps) than for the mechanisms of bibliometric exercise. Also, peer-review exercises typically occur over cycles of 5 to 6 years (the latest RAE covered an eight
…(Full text truncated)…
This content is AI-processed based on ArXiv data.
