Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics

Accepted for publication in the Journal of Informetrics 1 Do altmetrics point to the broader im pact o f research? An overview of benef its and disa dvantages of al tmetrics Lutz Bornmann Division for Science and I nnovation Studies Administrative Headquarters of the Max Planck Society Hofgartenstr. 8, 80539 Munich, Germany. Email: bornmann@gv.mpg.de 2 Abstract Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an interesting option for assessing the societal impact of research, as they offer new ways to measure (public) engagement with re search output. Altmetrics is a term to describe web-based metrics for the impact of publications and other scholarly material by using data from social media platforms (e.g. Twitter or Mendeley). This overview of studies explores the potential of a ltmetrics for measuring societal impact. It deals with the definition and classification of altmetric s. Furthermore, their benefits and disadvantages for measuring impact are discusse d. Keywords Social impact; Broader impact; Altmetrics; Scientometrics 3 1 Introduction Until a few decades ago, the general assumption in science policy was that a society can benefit most from research that is conducted at a very high level – evalu ated according to the standards inherent in science. In recent years, this automatistic approach has found less favour in science policy: policymakers expect science to demonstra te its value to society (Bornmann, 2013). A good example of this trend can be f ound in a recent book b y Bastow, Dunleavy, and Tinkler (2014), which is an attempt to “ re -explain the distinctive and yet more subtle ways in which the contemporary social sciences now shape and inform human development” (p. 2). The trend towards audit science is framed in a general change to the science landscape and is frequently described as a development from Mode 1 to Mode 2: While in Mode 1 science was characterized by the academic interests of a scientific community, Mode 2 is more concerned with the collaboration betwe en science and other areas of society and with research that is re levant to a particular application in society (Gibbons et al., 1994). It is not clear how the impact of research on other are as of society should be measured – unlike the impact which research has on itself. While peer re view and bibliometr ics have become standard methods for measuring the impact of research on other researc h, there is not yet an accepted framework within which to measure societal impact. Nowaday s, the case study approach to societal impact is favoured; however, this approach does not meet all the requirements generally associated with a societal impact framework. According to Frank and Nason (2009), the best method of measuring societal impact (in health research) should be “ feasible, not too labour intensive, and economically viable. It should be as accurate and responsive as possible within a reasonable evaluation budge t that should represent a small percentage of the money invested in the research being assessed ” (p. 531). There is a need for indicators which can reliably and validly measure the impact of research on certain parts of 4 society, with the primary aim of creating productive interac tion and successful communication between research a nd societal stakeholders. “ Scientists must be able to explain what they do to a broader public to garner political support and funding f or endeavours whose outcomes are unclear at best and dangerous a t worst, a difficult y which is magnified by the complexity of scientific issues” ( Puschmann, 2014, p. 91). 2 What are altmetrics? Alternative metrics (called altmetrics to distinguish them from bibliometrics, Gunn, 2013) are considered an interesting option for assessing the societa l impact of research, as they offer new ways to measure (public) e ngagement with research output ( Piwowar, 2013). “Altmetrics … is a term to describe web-based metrics for the impact of scholarly material, with an emphasis on social media outlets as sources of data ” (Shema, Bar-Ilan, & Thelwall, 2014). 1 In ‘ article-level metrics ’ (ALMs, Fenner, 2013b), views, downloads, clicks, notes, saves, tweets, shares, likes, recommends, tags, posts, trackbac ks, discussions, bookmarks, and comments are counted, rather than just citations of a pa per in a database such as Scopus (Elsevier), or by a publisher such a s the Public Library of Science (P LOS, Fenner, 2013b) (Liu, Xu, Wu, Chen, & Guo, 2013 ; Zahedi, Costas, & Wouters, 2014). Adie and Roe (2013) call these individual events (tweets or shares, for example) ‘ mentions ’ if they link to papers (and ‘ posts ’ if they do not). Every form of ALM involves log data which measures individual mentions over a certain period of time (Haustein, 2014). “ Today, for every single use of an electronic resource, the sy stem can record which resource was used, who used it, where that person was, when it was used, what type of request was issued, what type of rec ord it was, and from where the article was used” (Kurtz & Bollen, 2010, p. 4). The more or less frequent 1 Rousseau and Ye (2 013) have proposed “influmetrics” as a new name for this new for m of metrics. Cronin (2013) thinks that “co mplementar y metrics” is more app ropriate than “alternative metrics”. “Influmetrics” ha s the advantage agai nst “ complementar y metrics” and “ altmetrics ” that it do es not provoke the q uestion “ complementary or alternative to what?” 5 “ use ” of research output can either be seen as the direct impact of research or as evidence of “ real ” impact (Neylon, Willmers, & King, 2014). The importance of this alternative for m of metrics is indicated by one of the biggest multidisciplinary database providers, Elsevier, not only entering into partnership with Altmetric, a start-up tracking and analysing the online activity around scholarly literature, but also buying Mendeley, which combines a citation manager w ith a scholarly social net work (Roemer & Borchardt, 2013). Furthermore, according to Chamberlain (2013) and Piwowar and Priem (2013), scholars are already including altmetrics in publication lists in their CVs (in addition to citation impact measurements), c onferences on the subject are being arranged (such as altmetrics.org/altmetrics14) and organizations (such as ImpactStory and Altmetric) founded to collect and provide altmetrics (Fenner, 2013a). Against the background of this development Bornmann (2014) and Taylor (2013a) are talking about a revolution in scientometrics, Lin and Fenner (2013) about a new paradigm of research assessment and Kurtz and Bollen (2010) about a renaissance in bibliometrics with, notabl y, a new definition of the expression “ impact of science ” . According to Galloway, Pease, and Rauh (2013) “altmetrics is a fast - moving and dynamic area”. However, the use of alternative metrics to evaluate re search is not new. It has a long tradition in scientometrics with the analy sis of acknowledgements, patents, mentorships, news articles, and usage in syllabi (Priem, 2014). The use of the Internet for alternative metrics began with “ webometrics ” (or “ cybermetrics ” ) whereby the number of times a paper was mentioned on the web was counted (Roemer & Borchardt, 2012). These mentions were called “ web citations ” (Shema, Bar-Ilan, & Thelwall, in press). Several studies have investigated the relationship between web citations and traditional text citations finding moderate correlation s in most cases (see e.g. Kousha & Thelwall, 2007 ; Vaughan & Shaw, 2005 ; Vaughan & Shaw, 2008). 6 Later on, server download data for papers was analysed in order to measure (scientists ’ ) interest in papers (Gunn, 2013). The development of the Internet into social web – as a new social media platform – also led to new ways to measure impact. The social web is characterized by many applications which promote pa rticipation, interconnections, social interaction and user-generated content (Greenhow & Gleason, 2014 ; Weller & Peters, 2012). The user of content in the social web not only consumes but also provides it and comments on it (King et al., 2013). In scientometrics, the focus has been moving from web citation analysis (and the analysis of download data) towards social media usage analysis (Li, Thelwall, & Giustini, 2012), known currently as “ altmetrics ” . In recent years, the use of the following seven platforms in the social web as alternative metrics is of primary interest: “bookmarking , reference managers, recommendation service s, comments on articles, microbloggin g, Wikipedia, and blogging” (Priem & Hemminger, 2010). These platforms provide an insight into the research process, as the data, analyses, and results can be exchanged, stored and discussed (Fausto et al., 2012). However, alternative metrics are (still) greatly in flux, with new tools being considered as data sources and established tools losing their appeal as data sources (Darling, Shiffman, Côté, & Drew, 2013) . Over rece nt years, however, a number of tools have proved particularly suitable for alternative measurement: according to Fenner (2013b), for example, since June 2012 93% of PLOS Biology papers have been mentioned on Twitter. 3 How can alt metrics be classified? As there are now a number of social media tools which can be used as a source for altmetrics (see e.g. the long list of Claussen et al., 2013, p. 360), they have been classified by authors into various areas. This classification not only provides an overview of the dif ferent metrics, but also indicates the type of application for which each metric is suitable. Two 7 possible forms of classification used by ImpactStory and PLOS are described in Table 1. As the classified metrics are genera ll y AL Ms and not only altmetrics, the category “ cited ” – as in “ cited by scientists ” – is also listed. Table 1. The ALM classification of ImpactStory and PLOS (Lin & Fenner, 2013) Area Scholars Pu blic ImpactStory Viewed PDF downloads HTML downloads Saved CiteULike, Mendeley Delicious Discussed Science blogs, journal comments Blogs, Twitter, Facebook Recommended Citations by editorials Press article Cited Citations, full-text mentions Wikipedia mentions PLOS No distinction is made between scholars and public Viewed HTML/ PDF (PLOS or PubMed Central), XML (PL OS) Saved CiteULike, Mendeley Discussed NatureBlogs, ScienceSeeker, ResearchBlogging , PLOS Comments, Wikipedia, Twitter, Facebook Recommended F1000Prime Cited CrossRef, PubMed Central, Web of Science, Scopus Both classifications cover viewing, storing, discussing, recommending a nd citing papers (or other products of research) and therefore mirror the whole process of user engagement, from the first look at a paper to its citation in (scholarly ) literature and thus relate to the various dimensions of research impa ct (Ne ylon & Wu, 2009). In this process of user engagement, it is expected that the number of counts per pa per falls (from vi ewing to storing, discussing, recommending and citing ) and the significance of individual mentions 8 increases (with citations having the large st significance) (Kurtz & Bollen, 2010). According to Lin and Fenner (2013), only one person in 70 cites a paper that they have downloaded from PLOS in their own paper. With the exception of one type of AL M in Table 1, all the ALMs are assigned in the same way to the two classifications: While Wikipedia is included under “ cited ” by ImpactStory, PLOS has it in the “ discussed ” category. The two classifications differ significantly in that ImpactStory disting uishes between impact on scholars (scientific impact) and the public (societal impact). However, this distinction is sometimes quite artifici al : PDFs are not only downloaded by scholars and HTML versions not only by the public. There are specific advantages and disadvantages to each of the metrics listed in the table in their measurement of impact. For example, comments can allow valuable and rapid feedback to a paper; however, they are not given freque ntl y e nough on individual papers to be used validly as a metric (Neylon & Wu, 2009). Plum Analytics – a supplier of impact metrics similar to ImpactStory – also uses a classification similar to that in Table 1, with usage, captures, mentions, social media, a nd citations. Further classifications for altmetrics are as follows: Haustein and Peters (2012) distinguish between Web data (e.g. tweets, bookmarks and blog posts) and Web tools (e.g. social bookmarking systems and reference managers). While the various Web tools can be categorized as sharing services (e.g. YouTube and Flickr) and social bookmarking services (e.g. Delicious) (Haustein, 2014), the social networks (such as Facebook and Twitter) can be divided into informal and formal networks (Rodgers & Barbrow, 2013). Gunn (2013) designates some altmetrics as content-rich (e.g. blog posts or Wikipedia links) and others as plentiful or content-poor (e.g. tweets or Facebook ’s “like”). 9 4 What benefits do alt metrics offer ? The following list of the benefits of altmetrics is based on a categ orisation of the benefits named in the literature by Wouters and Costas (2012). In an overview of new forms of impact measurements, these authors identified four benefits that altmetrics has compared to traditional metrics: (1) Broadness: altmetric s measure impact beyond science. (2) Diversit y : altmetrics can measure the impact of scholarly pro ducts other than papers. (3) Speed: altmetrics permit impact to be measured shortly after the publication of a paper ( or the completion of other products). (4) Openness: as a rule, it is easy to obtain altmetric data. 4.1 Broadness Most comments on the benefits of altmetrics relate their potential for measuring the broader impact of research, that is, beyond science (Priem, Parra, Piwowar, & Waagmeester, 2011 ; Priem, Piwowar, & He mminger, 2012 ; Weller, Dröge, & Puschmann, 2011). I t is hoped that altmetrics can deliver more tra nsparent descriptions of the interest, usage and reach of scholarly products (Fausto, et al., 2012 ; Taylor, 2013a) and also more diverse and nuanced forms of impact analyses than traditional metrics, such as bibliometrics, permit (Waltman & Costas, 2014). Statements such as those by Mohammadi, Thelwall, Haustein, and Larivière (2014) are pertinent: “ A noticeable percentage of Clinical Medicine papers were read by people who are apparently not academics and this is an important issue because some articles could be useful in clinical practice even if they are not cited in the literature.” As citations only relate to the assessment by scientific authors of the research conducted by fellow researchers, altmetrics offer access to the opinions of a wide r audience, such as professionals, undergraduates, government and – as a whole – the interested general public (Adie, 2014 ; Hammarfelt, 2014). The potential to measure the flow of re search into society complies with the wishes of politicians, research organisations and funders, such as the br oader impact criteria required by 10 the US National Science Foundation (Chamberlain, 2013). Konkiel and Scherer (2013) propose altmetrics as a supplementary indicator of impact with which to justify budget increases and recruiting faculty for univer sit y trustees and state legislatures. Some authors, such as Bik and Goldstein (2013), ascribe the potential to measure the “ true ” or “ full ” impact of research to altmetrics. However, this is an exception in the literature and undoubtedly overstates the possibilities they offer. “ Hidden impact ” , the term used by Taylor (2013b), is more fitting, with its implication that altmetrics could reveal impac t which traditional indicators have hitherto been unable to reveal. According to Fenner (2013b), altmetrics allow the impact of research to be measured in more practical fields, and papers of general interest to be highlighted better than with citations. This covers for ms of impact, such as polic y change, and effects on clinical prac tice, technical applications, education, and health policies (Haustein, 2014 ; Haustein et al., 2014 ; Neylon, et al., 2014). 4.2 Diversity Altmetrics are not only more diverse in kinds of data (see above) and accordingly numbers of data sources (whereas for traditional citations only the cited references in journals serve as data source), but also allow for evaluation of a greater diversity of products, i.e., not just publications. Research funders, such as the US National Science Foundation, expect meanwhile not only publications but also other products to be given as the outcome of research in proposals. This new require ment should be understood as an indication that not only publications but also other forms of scholarly products play an important part in research evaluation now (Piwowar, 2013 ; Rousseau & Ye, 2013). With reference to evaluating these other products, the proviso imposed by the US National Science Foundation that onl y c itable and accessible products can “ count ” is crucial. These products might be datasets, software, copyrights, algorithms, grey literature, and slides (Zahedi, et al., 2014). Altmetrics now offer the opportunity to determine the impact of these products both in science – t hey are usually 11 under-represented in the citation record (Priem, 2014) – and beyond science (Galloway, et al., 2013). As well as measuring the impact of products, altmetrics can also be used to track a variety of scholarly activities such as teaching a nd service activities (Rodge rs & Barbrow, 2013). For example, the impact of course packs and reading lists or attendance a t online open courses (MOOCs) can be measured (Taylor, 2013a). 4.3 Speed One of the biggest disadvantages of citation counts in measuring impact is that a reliable and valid measurement can only be provided seve ral ye ars after publication (Wang, 2013). Altmetrics, on the other hand, permit the impact of a paper (or other products) to be measured just a few days or weeks after it ha s appeared (Haustein, Peters, Bar-Ilan, et al., 2014 ; Mohammadi & Thelwall, 2014). For example, the results of Maflahi and Thelwall (in press) suggest that papers tend to attract more Mendeley readers than citations initially, but that the situation reverses after several years. Relatively soon after pub lication, a paper is read, bookmarked, saved, annotated and discussed within academic circles and by the public (Priem, 2014 ; Rodgers & Barbrow, 2013). The prompt tweeting or blogging of researc h results can even assist scientists to secure prior it y for the r esults before they are submitted to a journal, on the basis of a preprint (Darling, et al., 2013). Many social web tools offer real-time ac cess to structured altmetric data via application programming interfaces (APIs) (Priem & Hemminger, 2010), with which the impact of a paper can be tracked at any time after publication. This real-time access can be used by scientists and others to track online activities on certain research topics of interest in orde r to obtain references to i mportant studies which have just been published (Priem, Tarabore lli, Groth, & Neylon, 2010). 4.4 Openness A major problem with the societal impact analy ses undertaken up to now has been the availability of data. While citation counts for impact measure ments in science are available in 12 multi-disciplinary databases (such as the Web of Science, Thomson Reuters, and Scopus), there has been no such easily accessible broad-based data for measuring societal impact. F or this reason, the case study approach was favoured for measuring societal impact, whereby it was only measured case-specific and not standardized (Bornmann, 2012, 2013). Altmetrics represent an interesting option for measuring societal impact instead of a case study. In particular, free access to this data through Web APIs, which allow immediate f eedback about a large publication set (Galloway, et al., 2013) means that data collection is less problematic (Thelwall, Haustein, Lariviere, & Sugimoto, 2013). Furthermore, altmetric data is today based on platforms with clearly defined boundaries and data ty pes, as is the case with Twitter or Mendeley (Priem, 2014), which facilitates the analysis of data and the interpretation of results. 5 What are the disadv antages of alt m etrics? It goes without saying that altmetrics have disadvantage s as well as advantages. The y share this characteristic with traditional metrics. Not everything that is cited ha s been read, and the relevant publications are not always cited in the correct place in a manuscript (Haustein, 2014). Furthermore, there are numerous different reasons why scientists cite a publication – and they are not always related to intellectual influence (Bornmann & Daniel, 2008). For Priem (2014), a lack of theory, ease of gaming, and possible biases ar e three limitations of altmetrics. The following discussion generalizes and adds to this list. 5.1 Commercialisation As commercial providers, many services in the social media (such as Twitter and Facebook), have a large stake in as many people as possible communicating as often as possible via their portals. For example, e-mails constantly draw the a ttention of the users of these portals to other potentially interesting users and content. A lack of communication or an 13 unwillingness to communicate could result in the portal ’ s failure to thrive commercially . So far, there have been no empirical studies to investigate how much bias this promotion of communication creates for altmetrics. This kind of commerc ialisation plays next to no part in traditional metrics, such as bibliometrics. Scientists are not encouraged to cite as much as possible. Although publication and citation figur es are made available in the Web of Science and Scopus, neither Thomson Reuters nor Elsevier pursues strategies to incre ase the number of publishing or citing authors. 5.2 Data quality There are a number of different aspects concerning data quality which could lead to a limitation of altmetrics: 1. Bias: As not everyone (in a city, a country , etc.) uses social media platforms, a measurement of impact always relates to a specific sample of pe ople who have mentioned a paper more or less frequently. I t is assumed that this sample has a systematic bias towards younger or more fad-embracing people (Priem, 2014) or towards those with a professional interest in research (Neylon, et al., 2014). As there are no accurate user statistics or sample descriptions for individual social media platforms, this bias cannot be quantifie d. 2. Target: Altmetric counts are freque ntl y made ava ilable as counts of all relevant mentions on a platform. However, more information about user groups who have had to do with a scientific paper is essential for a valid measurement of societal impact: Has impact been measure d in government documents or on social media comment sites (Liu & Adie, 2013)? This more specific description of the impact achieved is usually lacking nowadays. 3. Multiple versions: Publications often exist in different versions (for example as 14 altmetrics to measure their impact results in ambiguity and redundancy ( Liu & Adie, 2013). 4. Different meanings: Citations might be simple mentions or extensive discussions of a cited paper. Meaning is similarly expanded when applie d to social media conversations. These can be very technica l and detailed, or also consist merely of a simple mention (Neylon, et al., 2014 ; Tay lor, 2013b ). It would be desirable to have the different meanings taken into account in the measurement. 5. Measurement standards: Every scientist knows what is being measured with a citation count: the number of times a paper is listed in the references of subsequently published documents. In altmetrics it is often not clear what is being measured – even if the source for the metric is the same. The respec tive numbers can refer to different forms of engagement, as in the example g iven by Liu and Adie (2013) : “Quantification of the mentions of scholarly articles on Facebook could take into account either all or just public wall posts, and these posts might be further parsed into the number of wall posts with an article mention or the number of ‘ likes ’ and comments on that wall post. Each number emphasizes something different and thus paints a slightly different picture of engagement with an article on Facebook” (p. 32). 6. Mention standards: There are precise rules governing when, where and in which form papers are cited in a document (American Psy chological Association, 2010 ), although not all scientists comply with them. There are no similar rules applying to the various social media platforms (Taylor, 2013b). This means that many links to the research under discussion are included in the text in different ways or not at all (Neylon, et al., 2014). This makes it significantly more difficult to count mentions of papers on these platforms. Providers of altmetric data, such as 15 Altmetric, try to solve the problem with text-mining mechanisms (Liu & Adie, 2013). 7. Normalization: Citations are normalized to allow cross-field and cross-time comparisons of the impact of papers (Bornmann, Leydesdorff, & Mutz, 2013 ; Vinkler, 2010). As higher altmetric scores can be expected from newer pa pers and papers on certain topics (such as evolution or climate change) than for older papers and papers on other topics, altmetric data should also be normalized (Holmberg & Thelwall, 2014 ; Taylor, 2013a ; Thelwall, et al., 2013). Only normalized scores allow the (societal) impact of papers on different topics and from different periods to be compared (Torres-Salinas, Cabezas-Clavijo, & Jimenez-Contreras, 2013). Up to now, it has not been common to normalize altmetrics, but I mpactStor y alrea dy does so on the basis of percentiles (Chamberlain, 2013 ; Roemer & Borchardt, 2013). 8. Replication: Citation numbers from the Web of Science and Scopus can be replicated as a rule – if one takes into account that the numbers rise over time. The replication of altmetric data is difficult, as da ta providers change, become quickl y obsolete or changes are made to the service they offer (Haustein, Peters, Sugimoto, Thelwall, & Larivière, 2014 ; Thelwall, et al., 2013). Particularly when altmetrics are to be used to evaluate research, it is important that the results can be re plicated. Results of research evaluation often lead to critical discussions. 5.3 Missing Evidence The lack of evidence of altmetrics relate to the scarcity of sophisticated empirical studies on altmetrics. Meaningful results ca n onl y be obtained with applica tion o f advanced empirical methods. According to Haustein, Peters, Sugimoto, et al. (2014) “large -scale studies of altmetrics are rare, and systema tic evidence about the reliability, validity , and context of 16 these metrics is lacking” (p. 657). Many studies which hav e been conducted on altmetrics up to now employ inadequate methods. Samples are collected randomly or according to the ‘ snowball ’ principle; correlation coefficients are not interpreted in accordance with established guidelines; many statistica l tests on the same dataset are conducted without a correction to the alpha level (the Bonferroni correction, for example) a nd the statistical significance is used as the (sole) criterion for the importance of results (Bornmann & Williams, 2013). In a content analysis, of a blog for example, the relia bility of the assigned categories is not measured as a rule. I t is onl y possible to determine inter-coder reliability with a comparison of the categorizations by two coders. 5.4 Manipulation It is much easier to manipulate altmetrics than bibliometrics (Rousseau & Ye, 2013 ; Thelwall, et al., 2013). Regarding traditional metrics, there are reports that journals try to increase their impact with several citations in editorials or that Google Scholar can create citations with false papers (Delgado López-Cózar, Robinson-García, & Torres-Salinas, 2014). However, there are many more and different opportunities for manipulation with altmetrics that are much easier to carry out. “ In particular, since social websites tend to have no quality control and no formal process to link users to offline identities it would be easy to systematically generate high altmetric score s for any given researcher or set of articles” (Thelwall, 2014, p. 4). For example, Twitter mentions can be ge nerated through fake accounts and “ robot tweeting ” (Darling, et al., 2013 ; Liu & Adie, 2013). A possible measure to counter manipulation of altmetrics is the c ross-calibration of data from different sources in order to reveal suspicious patterns in a source (Priem & Hemminger, 2010). 17 6 Discussion The significance ascribed to alternative metrics for the evaluation of research fluctuates. While Haustein, Peters, Sugimoto, et al. (2014) already view these metrics as a fixed part of research evaluation, Cronin (2013) is sceptical: “ Neither Twitter mentions nor Facebook ‘ likes ’ are, for now at any rate, accepted curre ncies in the academic marketplace; you are not going to get promoted for having been liked a lot, though it may well boost your ego. A robust h- index, on the other hand, could work wonders f or your career” ( p. 1523). Even if there is no conclusive evidence of the significance of altmetrics for research evaluation, it is clear that research on a nd the use of altmetrics is becoming more and more popular and the (critical) discussions about possible applica tion scenarios are increasing (Peters, Beutelspacher, Maghferat, & Terliesner, 2012). One gains the impression thereby that altmetrics is not a short-lived object of study in the information sciences, but is establishing itself as a new subfield (Priem, et al., 2010). The growing significance of altmetrics (in the information sciences) is also noticeable from the number of overviews on research into the subject which look at the area from different perspectives (Bar-Ilan, Shema, & Thelwall, 2014 ; Galloway, et al., 2013 ; Haustein, 2014 ; Priem, 2014 ; Rodge rs & Barbrow, 2013 ; Torres-Salinas, et al., 2013 ; Wouters & Costas, 2012). In a survey of bibliometricians, around 86% of those surveyed said that they thought altmetrics had some potential for author or article evaluation. Compare d with paper downloads or views, for which 72% see some potential, the potential of typical altmetric platforms, such as blogs or bookmarks on reference managers, is given at round 35%, which, however, is significantly lower (Haustein, Peters, Bar-Ilan, et al., 2014). This stud y has also concluded that altmetrics offer grea t potential (and many expectations are associated with altmetrics); however, there are a number of problems which must be solved befor e it is used to evaluate research. According to Zahedi, et al. (2014) “the study of altmetrics is in its early 18 stage” and Taylor (2013b) says that “little is known about the intentional, motivational or experiential motives of the users” (p. 19). Many of the studies published so far ha ve merely calculated correlations between citations and altmetrics (see e.g. Eysenbach, 2011). However, little knowledge is gained from these studies. The cor relation on a medium level found b y most studies is hardly meaningful and can be interpreted fairly loosely. As t here is no interest in replacing traditional bibliometrics with altmetrics, research should not concentra te on the connectedness, but on specific differences between the two metrics (Darling, e t al., 2013): in how far can altmetrics – unlike the traditional metrics – measure the broader impact of research? The literature gives an abundance of issues with which researc h into altmetrics should concern itself in future. The white paper by the National Inf ormation Standards Organization (2014) in particular offers a number of recommendations as to which re search is important in this area. Lin and Fenner (2013) emphasize, for example, the development of sophisticated technologies to analyse the demographics of research users more accurately . There is great interest in finding out whether users are scholars or non-scholars, and how they are distributed geographically and what stage s the y have reached in their careers. Where the broader impact of research is concerned, it is much more important to learn who has used an actual re search product and why, than to simply know “how many” people ha ve in total. According to Priem, et al. (2010), in order to be able to answer the question of whether altmetrics measure impact ( “ or just empty buzz ” ), it should be compared with expert evaluations (Sud & Thelwall, 2014): Do altmetric counts corre late with the evaluations by experts of the soci et al impact of a paper? A good source of data for studying this correlation is F1000 Prime (Mohammadi & Thelwall, 2013). What is known as the Faculty of 1000 (F1000) peer review system is accordingly not an ex-ante assessment of manuscripts provided for publication in a journal, but an ex-post assessment of papers which have already been published in journals. The Faculty members also attach tags to the papers indicating their 19 relevance for science (e.g. “ new finding ” or “good for teaching” ). The ex -post assessments and tags can be used for the investiga tion of altmetrics. In scientometrics, procedures should be developed to detect and repair ga ming in altmetrics. Furthermore, attention should be given to the problem of r epresentativeness of altmetric data (Haustein, 2014): For example, if one would like to measure the impact of a research product on politics, one needs to know how strongly politics is represe nted on a certain social media platform. Fundamentally, it should be ensured that when a ltmetrics are used in research evaluation, they are in an informed peer review process, exactly like the traditional metrics. Results based on altmetrics must therefore not lead direc tly to decisions about research funding, but should be used to help experts to make decisions in a peer review proce ss (Bornmann, 2011 ; Rousseau & Ye, 2013). The traditional and alternative metrics should complement (and not replace) each other in an informed peer review process. 20 References Adie, E. (2014). The grey literature from an altmetrics perspective - opport unity and challenges. Research Trends (37), 23-25. Adie, E., & Roe, W. (2013). Altmetric: enriching scholarly content with article-level discussion and metrics. Learned Publishing, 26 (1), 11-17. doi: 10.1087/20130103. American Psychological Association. (2010). Publication manual of the American Psychological Association (6. ed.). Washington, DC, USA: American Psychological Association (APA). Bar-Ilan, J., Shema, H., & Thelwall, M. (2014). Bibliographic References in Web 2.0. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multi-dimensional indicators of performance (pp. 307-325). Cambridge, MA, USA: MIT Press. Bastow, S., Dunleavy, P., & Tinkler, J. (2014). The impact of the social sciences . London, UK: Sage. Bik, H. M., & Goldstein, M. C. (2013). An Introduction to Social Media for Scientists. PLoS Biol, 11 (4), e1001535. doi: 10.1371/journal.pbio.1001535. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45 , 199-245. Bornmann, L. (2012). Measuring the societal impact of research. EMBO Reports, 13 (8), 673- 676. Bornmann, L. (2013). What is societal impact of resea rch and how can it be assessed? A literature survey. Journal of the American Society of Information Science and Technology, 64 (2), 217-233. Bornmann, L. (2014). Is there currently a scientific re volution in scientometrics? Journal of the Association for Information Science and Technology, 65 (3), 647-648. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64 (1), 45-80. doi: 10.1108/00220410810844150. Bornmann, L., Ley desdorff, L., & Mutz, R. (2013). The use of percentiles and percentile rank classes in the analysis of bibliometric data: opportunities and limits. Journal of Informetrics, 7 (1), 158-165. Bornmann, L., & Williams, R. (2013). How to calculate the practica l significance of citation impact differences? An empirical example from evaluative institutional bibliometrics using adjusted predictions and marginal e ffects. J ournal of Informetrics, 7 (2), 562- 574. doi: 10.1016/j.joi.2013.02.005. Chamberlain, S. (2013). Consuming article-level metrics: observations and lessons. Information Standards Quarterly, 25 (2), 4-13. Claussen, J. E., Cooney, P. B., Defilippi, J. M., Fox, S. G., Glaser, S. M., Hawkes, E., . . . Steward, C. (2013). Science Communication in a Dig ital Age: Social Media and the American Fisheries Society. Fisheries, 38 (8), 359-362. doi: 10.1080/03632415.2013.816289. Cronin, B. (2013). The evolving indicator space ( iSpace). Journal of the American Society for Information Science and Technology, 64 (8), 1523-1525. doi: 10.1002/asi.23041. Darling, E. S., Shiffman, D., Côté, I. M., & Drew, J. A. (2013). The role of Twitter in the life cycle of a scientific publication. PeerJ PrePrints, 1 , e16v11. doi: 10.7287/peerj.preprints.16v1. Delgado López-Cózar, E., Robinson-García, N., & Torres-Salinas, D. (2014). The Google scholar experiment: How to index false papers and manipulate bibliometric indicators. 21 Journal of the Association for Information Science and Technology, 65 (3), 446-454. doi: 10.1002/asi.23056. Eysenbach, G. (2011). Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res, 13 (4), e123. Fausto, S., Machado, F. A., Bento, L. F. J., Ia marino, A., Nahas, T. R., & Munger, D. S. (2012). Research Blogging: Inde xing and Re gistering the Change in Science 2.0. PLoS ONE, 7 (12), e50109. doi: 10.1371/journal.pone.0050109. Fenner, M. (2013a). Letter from the guest content editor: altmetrics have come of age. Information Standards Quarterly, 25 (2), 3. Fenner, M. (2013b). What Can Article-Level Metrics Do for You? PLoS Biol, 11 (10), e1001687. doi: 10.1371/journal.pbio.1001687. Frank, C., & Nason, E. (2009). Health research: measuring the social, health and economic benefits. Canadian Medical Association Journal, 180 (5), 528-534. doi: Doi 10.1503/Cmaj.090016. Galloway, L. M., Pease, J. L., & Rauh, A. E. (2013). Introduction to Altmetrics for Science, Technology, Engineering, and Mathematics ( STEM) Librarians. Science & Technology Libraries, 32 (4), 335-345. doi: 10.1080/0194262X.2013.829762. Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge: the dynamics of science and research in contemporary societies . London, UK: SAGE. Greenhow, C., & Gleason, B. (2014). Social scholarship: Reconsidering scholarly prac tices in the age of social media. British Journal of Educational Technology, 45 (3), 392-402. doi: 10.1111/bjet.12150. Gunn, W. (2013). Social Signals Reflec t Academic Impact: What it Means When a Scholar Adds a Paper to Mendeley . Information Standards Quarterly, 25 (2), 33-39. Hammarfelt, B. (2014). Using a ltmetrics for assessing research impact in the humanities. Scientometrics , 1-12. doi: 10.1007/s11192-014-1261-3. Haustein, S. (2014). Readership metrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multi-dimensional indicators of performance (pp. 327-344). Cambridge, MA, USA: MIT Press. Haustein, S., & Peters, I. (2012). Using social bookmarks and tags as alternative indicators of journal content description. firstmonday, 17 (11). Haustein, S., Peters, I., Bar-Ilan, J., Priem, J., Shema, H., & Terliesner, J. (2014). Coverage and adoption of altmetrics sources in the bibliometric community . Scientom etrics , 1- 19. doi: 10.1007/s11192-013-1221-3. Haustein, S., Peters, I., Sugimoto, C. R., Thelwall, M., & Larivière, V. (2014). Tweeting biomedicine: An analysis of tweets and citations in the biomedical literature. Journal of the Association for Information Science and Technology, 65 (4), 656-669. doi: 10.1002/asi.23101. Holmberg, K., & Thelwall, M. (2014). Disciplinary differences in Twitter scholarl y communication. Scientometrics , 1-16. doi: 10.1007/s11192-014-1229-3. King, D., Ramirez-Cano, D., Greaves, F., Vlaev, I., Beales, S., & Darzi, A. (2013). Twitter and the health reforms in the English National Health Service. Health Policy, 110 (2 - 3), 291-297. doi: 10.1016/j.healthpol.2013.02.005. Konkiel, S., & Scherer, D. (2013). New opportunities for repositories in the age of altmetrics. Bulletin of the American Society for Information Science and Technology, 39 (4), 22- 26. doi: 10.1002/bult.2013.1720390408. Kousha, K., & Thelwall, M. (2007). Google Scholar citations and Google Web/URL c itations: a multi-discipline exploratory anal ysis. Journal of the American Society for Information Science and Technology, 58 (7), 1055-1065. 22 Kurtz, M. J., & Bollen, J. (2010). Usage Bibliometrics. Annual Review of Information Science and Technology, 44 , 3-64. Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91 (2), 461-471. doi: 10.1007/s11192- 011-0580- x. Lin, J., & Fenner, M. (2013). Altmetrics in Evolution: Defining and Redefining the Ontology of Article- Level Metrics”,, Vol. 25, No. 2, pp. 2 0, Available at: 10.3789/isqv25no2.2013.04. Information Standards Quarterly, 25 (2), 20-26. Liu, C. L., Xu, Y. Q., Wu, H., Chen, S. S., & Guo, J. J. (2013). Correlation and Interaction Visualization of Altmetric Indica tors Extracted From Scholarly Social Network Activities: Dimensions and Structure. Journal of Medical Internet Research, 15 (11), 17. doi: 10.2196/jmir.2707. Liu, J., & Adie, E. (2013). Five c hallenges in altmetrics: A toolmaker's perspective. Bulletin of the American Society for Information Science and Technology, 39 (4), 31-34. doi: 10.1002/bult.2013.1720390410. Maflahi, N., & Thelwall, M. (in press). When are readers as good as citers for bibliometrics ? Scopus vs. Mendeley for LIS journals. Journal of the Association for Information Science and Technology . Mohammadi, E., & Thelwall, M. (2013). Assessing non-standard article impact using F1000 labels. Scientometrics, 97 (2), 383-395. doi: 10.1007/s11192-013-0993-9. Mohammadi, E., & Thelwall, M. (2014). Mendeley readership altmetrics for the social sciences and humanities: Researc h evaluation and knowledge flows. Journal of the Association for Information Science and Technology , n/a-n/a. doi: 10.1002/asi.23071. Mohammadi, E., Thelwall, M., Haustein, S., & Larivière, V. (2014). Who Reads Research Articles? An Altmetrics Analy sis of Mendeley User Categories1. Retrieved March 18, from http://www.academia.edu/6298635/Who_Reads_Researc h_Articles_An_Altmetrics_A nalysis_of_Mendeley_User_Categories National Information Standards Organization. (2014). Altmetrics standards project white paper . Baltimore, MD, USA: National Information Standards Organization (NISO). Neylon, C., Willmers, M., & King, T. (2014). Rethinking Impact: Applying Altmetrics to Southern African Research . Ottawa, Canada: International Development Research Centre. Neylon, C., & Wu, S. (2009). Article-level metrics and the evolution of scientific impact. Plos Biology, 7 (11). doi: e1000242 10.1371/journal.pbio.1000242. Peters, I., Beutelspacher, L., Mag hferat, P., & Terliesner, J. (2012). Scientific bloggers under the altmetric microscope. Proceedings of the American Society for Information Science and Technology, 49 (1), 1-4. doi: 10.1002/meet.14504901305. Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493 (7431), 159-159. Piwowar, H., & Priem, J. (2013). The power of altmetrics on a CV. Bulletin of the American Society for Information Science and Technology, 39 (4), 10-13. doi: 10.1002/bult.2013.1720390405. Priem, J. (2014). Altmetrics. I n B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multi-dimensional indicators of performance . Cambridge, MA, USA: MIT Press. Priem, J., & Hemminger, B. M. (2010). Scientometrics 2.0: toward ne w metrics of scholarly impact on the social Web. First Monday, 15 (7). Priem, J., Parra, C., Piwowar, H., & Waagmeester, A. (2011). Uncovering impacts : CitedIn and total-impact, two new tools for gathering altmetrics . Paper presented at the iConference 2012. 23 Priem, J., Piwowar, H., & Hemminger, B. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. Retrieved March 27, from Priem, J., Taraborelli, D., Groth, P., & Ney lon, C. (2010). Altmetrics: a manifesto. Retrieved March 28, from http://altmetrics.org/manifesto/ Puschmann, C. (2014). (Micro)Blogging Science? Notes on Potentials and Constraints of New Forms of Scholarly Communication. In S. Bartling & S. Friesike (Eds.), Opening Science (pp. 89-106): Springer International Publishing. Rodgers, E. P., & Barbrow, S. (2013). A look at altmetrics and its growing significance to research libraries . Ann Arbor, MI, USA: The University of Michigan University Library. Roemer, R. C., & Borchardt, R. (2012). From bibliometrics to altmetrics: A changing scholarly landscape. Co llege & Research Libraries News, 73 (10), 596-600. Roemer, R. C., & Borchardt, R. (2013). Institutional altmetrics & academic libraries. Information Standards Quarterly, 25 (2), 14-19. Rousseau, R., & Ye, F. Y. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 58 . doi: 10.1007/s11434-013-5939-3. Shema, H., Bar-Ilan, J., & Thelwall, M. (2014). Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics. Journal of the Association for Information Science and Technology, 65 (5), 1018-1027. doi: 10.1002/asi.23037. Shema, H., Bar-Ilan, J., & Thelwall, M. (in press). How is research blogged? A content analysis approach. Journal of the Association for Information Science and Technology . Sud, P., & Thelwall, M. (2014). Evaluating altmetrics. Scientometrics, 98 (2), 1131-1143. doi: 10.1007/s11192-013-1117-2. Taylor, M. (2013a). Exploring the boundarie s: how altmetrics can expand our vision of scholarly communication and social impact. Information Standards Quarterly, 25 (2), 27-32. Taylor, M. (2013b). Towards a c ommon model of citation: some thoughts on merging altmetrics and bibliometrics. Research Trends (35), 19-22. Thelwall, M. (2014). A brief history of altmetrics. Research Trends (37), 3-4. Thelwall, M., Haustein, S., Lariviere, V., & Sugimoto, C. R. (2013). Do Altmetrics Work? Twitter and Ten Other Social Web Services. Plos One, 8 (5). doi: 10.1371/journal.pone.0064841. Torres-Salinas, D., Cabezas-Clavijo, A., & Jimenez-Contreras, E. (2013). Altmetrics: new indicators for scientific communication in Web 2.0. Comunicar, 41 , 53-60. Vaughan, L., & Shaw, D. (2005). Web citation data for impac t assessment: A comparison of four science disciplines. Journal of the American Society for Information Science and Technology, 56 (10), 1075-1087. doi: Doi 10.1002/Asi.20199. Vaughan, L., & Shaw, D. (2008). A new look at evidence of scholarly c itation in citation indexes and from web sources. Scientometrics, 74 (2), 317-330. doi: DOI 10.1007/s11192-008-0220-2. Vinkler, P. (2010). The evaluation of research by scientometric indicators . Oxford, UK: Chandos Publishing. Waltman, L., & Costas, R. (2014). F1000 Rec ommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations. Journal of the Association for Information Science and Technology, 65 (3), 433-445. doi: 10.1002/asi.23040. Wang, J. (2013). Citation time window choice for research impact evaluation. Scientometrics, 94 (3), 851-872. doi: 10.1007/s11192-012-0775-9. Weller, K., Dröge, E., & Puschmann, C. (2011). Citation analy sis in Twitter: approaches for defining and measuring information flows within Tweets during scie ntific 24 conferences. In M. Rowe, M. Stankovic, A.-S. Dadzie & M. Hardey (Eds.), Making Sense of Microposts (MSM2011) (pp. 1-12). Heraklion, Crete: CEUR workshop proceedings. Weller, K., & Peters, I. (2012). Citations in Web 2.0. I n A. Tokar, M. Beurskens, S. Keuneke, M. Mahrt, I. Peters, C. Puschmann, T. van Treeck & K. Weller ( Eds.), Scie nce and the Internet (pp. 209-222). Düsseldorf. Germany: Düsseldorf University Press. Wouters, P., & Costas, R. (2012). Users, narcissism and control – tracking the impact of scholarly publications in the 21st century . Utrecht, The Netherlands: SURFfoundation. Zahedi, Z., Costas, R., & Wouters, P. (2014). How well deve loped are altmetrics? A cross - disciplinary analysis of the pre sence of ‘alternative metrics’ in scientific publications. Scientometrics , 1-23. doi: 10.1007/s11192-014-1264-0.