Recommender Systems for the Conference Paper Assignment Problem

Recommender Systems for the Conference Paper Assignment Problem Don Conry†, Yehuda Koren$, and Naren Ramakrishnan† †Department of Computer Science, Virginia Tech, VA 24061, USA $Yahoo! Research, Israel ABSTRACT Conference paper assignment, i.e., the task of assigning pa- per submissions to reviewers, presents multi-faceted issues for recommender systems research. Besides the traditional goal of predicting ‘who likes what?’, a conference manage- ment system must take into account aspects such as: re- viewer capacity constraints, adequate numbers of reviews for papers, expertise modeling, conflicts of interest, and an over- all distribution of assignments that balances reviewer pref- erences with conference objectives. Among these, issues of modeling preferences and tastes in reviewing have tradition- ally been studied separately from the optimization of paper- reviewer assignment. In this paper, we present an integrated study of both these aspects. First, due to the paucity of data per reviewer or per paper (relative to other recommender systems applications) we show how we can integrate multi- ple sources of information to learn paper-reviewer preference models. Second, our models are evaluated not just in terms of prediction accuracy but in terms of the end-assignment quality. Using a linear programming-based assignment opti- mization formulation, we show how our approach better ex- plores the space of unsupplied assignments to maximize the overall affinities of papers assigned to reviewers. We demon- strate our results on real reviewer preference data from the IEEE ICDM 2007 conference. Categories and Subject Descriptors H.4.2 [Information Systems Applications]: Types of Systems—Decision support; J.4 [Computer Applications]: Social and Behavioral Sciences Keywords Recommender systems, collaborative filtering, conference pa- per assignment, linear programming. 1. INTRODUCTION Modern conferences, especially in areas such as data min- ing/machine learning (KDD; ICDM; ICML; NIPS) and da- tabases/web (VLDB; SIGMOD; WWW), are beset with ex- cessively high numbers of paper submissions. Assigning these papers to appropriate reviewers in the program com- mittee (which can constitute a few hundred members) is a herculean task and hence motivates the use of recommender systems. Besides the traditional goal of predicting ‘who likes what?’, a conference management system must take into account as- pects such as: reviewer capacity constraints, adequate num- bers of reviews for papers, expertise modeling, conflicts of interest, and an overall distribution of assignments that bal- ances reviewer preferences with conference objectives. Among these, issues of modeling preferences, expertise, and tastes in reviewing have traditionally been studied separately from the optimization of paper-reviewer assignment. The former has been the subject of much academic research (see Sec- tion 2.1) while the latter is emphasized by commercial soft- ware, such as EasyChair, CyberChair, and Microsoft’s CMS, which aim to automate the management of the conference reviewing process. We investigate the conference paper assignment problem (CPAP) through the lens of recommender systems research. There are three key differences from traditional recommender systems research and the CPAP problem. First, in a tradi- tional recommender, recommendations that meet the needs of one user do not affect the satisfaction of other users. In CPAP, on the other hand, multiple users (reviewers) are bid- ding to review the same papers and hence there is the pos- sibility of one user’s recommendations (assignments) affect- ing the satisfaction levels (negatively) of other users. Hence the design of reviewer preference models must be posed and studied in an overall optimization framework. Second, in a conventional recommender, the goal is often to recommend new entities that are likely to be of interest, whereas in CPAP, the goal is to ensure that reviewers are predominantly assigned their (most) preferred papers. Nev- ertheless, preference modeling is still crucial because it gives the assignment algorithm some degree of latitude in aiming to satisfy multiple users. Finally, recommender systems are used to working with sparse data but the amount of ‘signal’ available to model preferences in the CPAP domain is exceedingly small; hence we must integrate multiple sources of information to build strong preference models. In this paper, we present the first integrated study of both modeling reviewing preferences and optimizing assignments for conference management. Our key contributions can be summarized as follows. 1. Due to the paucity of data per reviewer or per pa- per (relative to other recommender systems applica- tions) we show how we can integrate information about publication subject categories, contents of paper ab- stracts, and co-authorship information to learn im- proved paper-reviewer preference models. 2. We evaluate our models not just in terms of prediction a

…(Full text truncated)…

This content is AI-processed based on ArXiv data.