Comment: Bibliometrics in the Context of the UK Research Assessment Exercise

arXiv:0910.3532v1 [stat.ME] 19 Oct 2009 Statistical Science 2009, Vol. 24, No. 1, 15–16 DOI: 10.1214/09-STS285A Main article DOI: 10.1214/09-STS285 c ⃝Institute of Mathematical Statistics, 2009 Comment: Bibliometrics in the Context of the UK Research Assessment Exercise Bernard W. Silverman Abstract. Research funding and reputation in the UK have, for over two decades, been increasingly dependent on a regular peer-review of all UK departments. This is to move to a system more based on bib- liometrics. Assessment exercises of this kind influence the behavior of institutions, departments and individuals, and therefore bibliometrics will have effects beyond simple measurement. Key words and phrases: Bibliometrics, research funding, perverse in- centives. In the United Kingdom’s Research Assessment Ex- ercise (RAE), every university may submit its re- search in every discipline for assessment. On this assessment rests a considerable amount of funding; indeed a number of universities, leading “research universities” in American nomenclature, gain more from this source of research funding than from gov- ernment funding for teaching. Within broad subject bands, the Higher Education Funding Council for England funds teaching on a flat rate per student. So the amount of funding a student of Mathemat- ics attracts is the same whichever university they attend. On the other hand, funding for research is selective: those departments which fare well on the Research Assessment Exercise receive more funding as a result. This is in addition to any income from grants and grant overheads. The RAE and its predecessors have been running for over two decades, and have always been based on peer review, though numerical data on student numbers and grant income also have some input into the assessment. However, it is proposed that “met- rics,” which include so-called bibliometric data, will B. W. Silverman is Master, St Peter’s College, Oxford OX1 2DL, United Kingdom e-mail: bernard.silverman@spc.ox. ac.uk. This is an electronic reprint of the original article published by the Institute of Mathematical Statistics in Statistical Science, 2009, Vol. 24, No. 1, 15–16. This reprint differs from the original in pagination and typographic detail. be the main part of the system which will soon suc- ceed the RAE, though it is probable that in mathe- matical subjects, peer review will continue to play a considerable part. The details have yet to be worked out. In the 2008 RAE, I was chair of the committee which reviewed Probability, Statistics and the more mathematical aspects of Operational Research. The committee’s experience of conducting the assessment as a whole strengthened our view that peer review must be at the core of any future assessment of research in our area. Reliance on bibliometric and purely quantitative methods of assessment would, in our unanimous view, introduce serious biases, both into the assessment process and, perhaps more se- riously, into the behavior of institutions and of in- dividual researchers, to the detriment of the very research which the exercise is intended to support. It is important to stress the effect of any system on the behavior of institutions. The current peer-review RAE has had clear effects on institutional behavior, some of them certainly positive, some of them per- haps less so. For example, the RAE gives explicit advantages to new entrants to the profession; those entering in the last few years are allowed to submit a smaller corpus of work for assessment, and there is also credit given within the peer review system for a subjective assessment of the general vitality of the department. Of the approximately 400 research- active faculty declared to the statistics panel in the 2008 RAE, about a quarter were new entrants since 2001, and the RAE has certainly given an impetus 1 2 B. W. SILVERMAN to this new recruitment, as it also does to the mo- bility of leading researchers between institutions. On a more negative note, the fixed date of the assess- ment encourages a “boom-bust” mentality, where some institutions hire in considerable numbers of new faculty in the period leading up to the census date; to make up for this extra expenditure, during the period after the census date there is something of a moratorium on appointments. The consideration of grant income in the RAE gives extra gearing to the pressure on faculty to pursue grant-supported research rather than to work in a more individual fashion. There can be little doubt that a stronger empha- sis on bibliometrics (and other “metrics”) in assess- ment exercises will affect institutional behavior, es- pecially in systems where assessment results have both reputational and fiscal impact. Because indi- viduals are sensitive to institutional pressures, they too will modify their behavior in response. For ex- ample, it is probably the case that there is a high correlation between h-index (say) and perceived qual- ity and reputation of researchers.

This content is AI-processed based on open access ArXiv data.