Usage Bibliometrics as a Tool to Measure Research Activity

Usage Bibliometrics as a Tool to Measure Research Activity 
 Edwin A. Henneken, Michael J. Kurtz
 Smithsonian Astrophysical Observatory, 60 Garden Street, Cambridge, MA 02138, USA
 
 Abstract Measures for research activity and impact have become an integral ingredient in the assessment of a wide range of entities (individual researchers, organizations, instruments, regions, disciplines). Traditional bibliometric indicators, like publication and citation based indicators, provide an essential part of this picture, but cannot describe the complete picture. Since reading scholarly publications is an essential part of the research life cycle, it is only natural to introduce measures for this activity in attempts to quantify the efficiency, productivity and impact of an entity. Citations and reads are significantly different signals, so taken together, they provide a more complete picture of research activity. Most scholarly publications are now accessed online, making the study of reads and their patterns possible. Click-stream logs allow us to follow information access by the entire research community, real-time. Publication and citation datasets just reflect activity by authors. In addition, download statistics, derived from these click-streams, will help us identify publications with significant impact, but which do not attract many citations. Click-stream signals are arguably more complex than, say, citation signals. For one, they are a superposition of different classes of readers. Systematic downloads by crawlers also contaminate the signal, as does random browsing behavior. We will discuss the complexities associated with clickstream data and how, with proper filtering, statistically significant relations and conclusions can be inferred from download statistics. We will describe how download statistics can be used to describe research activity at different levels of aggregation, ranging from organizations to countries. These statistics show a strong correlation with socio-economic indicators, like the GDP. A comparison will be made with traditional bibliometric indicators. Since we will be using click-stream data from the Astrophysics Data System (ADS), we will argue that astronomy is representative of more general trends.
 !1 Introduction 3 Definition of Terminology 5 Usage and Research Activity 9 Traditional Indicators 15 Discussion 16 Concluding Remarks 20 Acknowledgements 21 Bibliography 22 Author Information 25 !2 
 Introduction The standard indicators to measure the quality and quantity of scholarly research are funds expended, number of papers published, and number of citations to those papers and measures derived from these citations. Since the turn of the century a fourth key indicator for research assessment has arisen: measures of the use of the (now almost exclusively) digital research documents. The concept of “research documents” represents a more general class of expressions of research than “scholarly publications”, because it contains anything that may get published during research. This study is confined to scholarly publications, that is articles in scholarly journals. Scholarly research can be represented by identifying a “research cycle”; research consists of activity expended in the various stages of this cycle (illustrated in figure 1). We assume that taking the publication stage of the research cycle as a proxy will sufficiently represent “research activity”. Also, from a practical point of view, the publication stage is the only stage in this cycle that allows for clearly quantifiable metrics. We therefore focus on usage of scholarly publications. !3 !

Figure 1: graphical representation of research cycle The use of this usage information is in its infancy. The leading assessments of the quality and quantity of research on a country basis (Science & Engineering Indicator [1], for a given year), a university basis (Times Higher Education [2] or ARWU [3] ranking, for a given year), or on a journal basis (Impact Factor [4], Eigenfactor [5,6] or Source Normalized Impact per Paper [7], for a given year) do not use usage information. The only widespread use of digital download records is by librarians making purchase decisions, aided by the COUNTER [8] standard for a given year, continuing their practice from the print era. Since the first obsolescence function based on digital downloads was published 20 years ago [9] there has been an avalanche of work on the nature of digital download information. The review by Kurtz and Bollen [10] contains 171 references, more recent work with extensive discussions and bibliographies include [11, 12, 13, 14, 15, 16, 17, 18]. Download information is now commonly found on article pages at journal web sites and on various aggregator web sites. Perhaps the most influential of these are the article download counts for a given year by the SSRN. Before usage information can be widel

…(Full text truncated)…

This content is AI-processed based on ArXiv data.