The response by Benedetto, Checchi, Graziosi & Malgarini (2017) (hereafter "BCG&M"), past and current members of the Italian Agency for Evaluation of University and Research Systems (ANVUR), to Franceschini and Maisano's ("F&M") article (2017), inevitably draws us into the debate. BCG&M in fact complain "that almost all criticisms to the evaluation procedures adopted in the two Italian research assessments VQR 2004-2010 and 2011-2014 limit themselves to criticize the procedures without proposing anything new and more apt to the scope". Since it is us who raised most criticisms in the literature, we welcome this opportunity to retrace our vainly "constructive" recommendations, made with the hope of contributing to assessments of the Italian research system more in line with the state of the art in scientometrics. We see it as equally interesting to confront the problem of the failure of knowledge transfer from R&D (scholars) to engineering and production (ANVUR's practitioners) in the Italian VQRs. We will provide a few notes to help the reader understand the context for this failure. We hope that these, together with our more specific comments, will also assist in communicating the reasons for the level of scientometric competence expressed in BCG&M's heated response to F&M's criticism.
The response by Benedetto, Checchi, Graziosi & Malgarini (2017) (hereafter "BCG&M"), past and current members of the Italian Agency for Evaluation of University and Research Systems (ANVUR), to Franceschini and Maisano's ("F&M") article (2017), inevitably draws us into the debate. BCG&M in fact complain "that almost all criticisms to the evaluation procedures adopted in the two Italian research assessments VQR 2004-2010 and 2011-2014 limit themselves to criticize the procedures without proposing anything new and more apt to the scope". Since it is us who raised most criticisms in the literature, we welcome this opportunity to retrace our vainly "constructive" recommendations, made with the hope of contributing to assessments of the Italian research system more in line with the state of the art in scientometrics. We see it as equally interesting to confront the problem of the failure of knowledge transfer from R&D (scholars) to engineering and production (ANVUR's practitioners) in the Italian VQRs. We will provide a few notes to help the reader understand the context for this failure. We hope that these, together with our more specific comments, will also assist in communicating the reasons for the level of scientometric competence expressed in BCG&M's heated response to F&M's criticism.
ANVUR began operations in May 2011, with the appointment of seven full-time members of the executive committee, following public competition for the positions (full disclosure: one of us applied unsuccessfully to the original 2010 call, as well as the 2015 call for renewal of committee members).2 All members of the 2011 committee were professors (one each in physics, engineering, medicine, veterinary sciences, sociology, two in economic sciences). At that time, only one (Andrea Bonaccorsi) had ever authored a Scopus or WoS indexed publication on research evaluation. Professor Bonaccorsi, an economist by education, showed marginal diversification of his scientific production in research evaluation. A managing director was hired soon after the executive committee assumed operations, followed by several staff members and collaborators on temporary contracts. On 7 November 2011, ANVUR, acting on authority of the Italian Ministry of Education, University and Research (MIUR), launched the 2004-2010 VQR. Unfortunately, five months from appointment would likely have been too little for adequate conception, design and organization of such a large scale exercise, even for an executive committee composed of seven of the world’s most experienced scientometricians. For the first VQR, Sergio Benedetto (one of the authors of the response to F&M), a professor in telecommunications engineering, was appointed as VQR Coordinator, with Andrea Bonaccorsi as Assistant Coordinator. When the second VQR (2011-2014) was launched (3 September 2015) the executive committee had for the most part been newly appointed, with only Sergio Fantoni, a physicist (president of the original committee), and Sergio Benedetto held over. All five new entries were again professors: Andrea Graziosi (an author of the response to F&M), professor in contemporary history, succeeded Sergio Fantoni as Committee President; Daniele Checchi (also a responder to F&M), a professor in economics, succeeded Sergio Benedetto as VQR Coordinator. The remaining members of the current committee are a neuroscientist, a mathematician, and a surgeon. None of the new entries shows a publication on research evaluation in indexed journals. Finally, the last author of the response to F&M, Marco Malgarini, joined ANVUR staff in 2012 and is now head of the research evaluation area. At the time of hiring he had no indexed publications on research evaluation, but now counts three.
Before dealing with the “constructive” criticism issue raised by BCG&M, we would like to comment on their specific responses to F&M.
In Section 2, paragraph 7 of their response, BCG&M mention two publications as heavily critical to the 2004-2010 VQR. Because these are by Italian authors, BCG&M dispute F&M’s consideration that such criticism is “international”, with the inference that this also makes it of little significance. Our view is that science has no nationality, and that good science does not depend on the authors’ location, rather on the truths that it hopefully embeds. Perhaps F&M appreciated the “international” standing of the criticism in the sense of the global context of the hosting journals. (There have also been more than two critical articles by Italian authors, all published in international bibliometric journals, as noted in Section 5.)
In Section 3.1, BCG&M state: “To show the inconsistency of this exercise suffices to say that the VQR 2011-2014 results published on February 22 nd , 2017, show that only 32.6% of the proposed products were assessed as excellent, and a total of 63.4 are in the two classes A or B.” We do not enter into the “triviality” of F&M’s critique, as labeled by BCG&M. Whether t
This content is AI-processed based on open access ArXiv data.
