Gender, Productivity, and Prestige in Computer Science Faculty Hiring Networks

Gender, Pro ductivit y , and Prestige in Computer Science F acult y Hiring Net w orks Sam uel F. W ay, 1 , ∗ Daniel B. Larremore, 2 , † and Aaron Clauset 1, 3, 2 , ‡ 1 Dep artment of Computer Scienc e, University of Color ado, Boulder CO, 80309 USA 2 Santa F e Institute, Santa F e NM, 87501 USA 3 BioF r ontiers Institute, University of Color ado, Boulder CO, 80303 USA W omen are dramatically underrepresented in computer science at all levels in academia and ac- coun t for just 15% of tenure-trac k faculty . Understanding the causes of this gender imbalance w ould inform b oth p olicies intended to rectify it and employmen t decisions by departmen ts and individuals. Progress in this direction, how ev er, is complicated by the complexity and decentralized nature of faculty hiring and the non-indep endence of hires. Using comprehensiv e data on both hiring outcomes and scholarly pro ductivity for 2659 ten ure-track faculty across 205 Ph.D.-granting departmen ts in North America, we inv estigate the multi-dimensional nature of gender inequality in computer science facult y hiring through a net w ork mo del of the hiring process. Overall, w e find that hiring outcomes are most directly affected b y (i) the relative prestige betw een hiring and placing institutions and (ii) the scholarly pro ductivit y of the candidates. After including these, and other features, the addition of gender did not significantly reduce mo deling error. How ever, gender differ- ences do exist, e.g., in scholarly pro ductivit y , p ostdo ctoral training rates, and in career mov ements up the rankings of universities, suggesting that the effects of gender are indirectly incorp orated in to hiring decisions through gender’s cov ariates. F urthermore, we find evidence that more highly rank ed departments recruit female faculty at higher than exp ected rates, whic h app ears to inhibit similar efforts by low er rank ed departmen ts. These findings illustrate the subtle nature of gender inequalit y in faculty hiring netw orks and provide new insights to the underrepresentation of women in computer science. Keywords: net work analysis, mo deling, gender, so cial dynamics, employmen t networks, data science I. INTR ODUCTION W omen contin ue to b e dramatically underrepresen ted in computer science, receiving only 18% of bachelors’ degrees and 20% of do ctorates in 2011, 1 and are esti- mated to hold fewer than 20% of technical p ositions in the computing industry . 2 W omen are esp ecially under- represen ted in the professoriate, making up only 15% of ten ured or ten ure-track faculty in computer science de- partmen ts [1]. Understanding the causes of gender im- balance in faculty hiring w ould illuminate the underlying so cial pro cesses that shape academic disciplines, and fa- cilitate efforts b oth to supp ort equal opp ortunities and to address the many non-merito cratic differences in male and female facult y exp eriences [2 – 4]. These differences include disparities in ten ure rates, comp etency ev alua- tions, remuneration, allo cation of researc h facilities, and gran t comp etitions. Rectifying these differences and im- pro ving the gender balance in computer science would serv e not only to adv ance so cial justice but would also promote the sort of diversit y in skills and research ap- proac hes that has b een found to improv e group p erfor- mance [5], particularly in innov ation-fo cused industries [6]. ∗ samuel.w ay@colorado.edu † larremore@santafe.edu ‡ aaron.clauset@colorado.edu 1 http://nces.ed.gov/programs/digest/2013menu_ tables.asp 2 http://cnet.co/1GZh268 Muc h of the past researc h on gender imbalance among facult y has fo cused on the “leaky pip eline,” the name giv en to the observ ation that women leav e science, tech- nology , engineering and mathematics (STEM) fields at greater rates than men at every stage of an academic career, from grade sc ho ol to full professor [7]. At the facult y hiring stage of the pip eline, several exp erimen- tal studies ha ve aimed to identify the causes of gender im balance [8 – 10]. How ever, these hav e yielded inconsis- ten t, ev en con tradictory findings, and little past w ork has fo cused sp ecifically on computer science. Essen tially , faculty hiring is a communit y-based com- p etitiv e pro cess of sub jective expert ev aluations un- der conflicting and ev olving preferences; that is to say , it’s complicated. These features, along with the non- indep enden t nature of hiring outcomes, mak e it difficult to reliably assess the presence and source of real biases. Here we in vestigate the role of gender in faculty hiring in computer science using a nov el netw ork mo del of the hir- ing process itself, across institutions and time. W e then use this mo del to study the hiring histories of individ- ual institutions and the exp eriences of individual faculty . W e train this mo del using comprehensiv e data on the hir- ing outcomes, scholarly pro ductivity , and gender of 2659 ten ured or ten ure-track faculty across all 205 computer science Ph.D.-granting departments in the United States and Canada [1]. Man y studies ha ve found evidence of gender bias in academia. F or instance, male faculty in the life sci- ences tend to train fewer female graduate students and p ostdocs, relative to female representation in the po ol of trainees [11]. This tendency is more pronounced at elite 2 institutions, whic h tend to pro duce the ma jorit y of future facult y [1]. W omen often p erceive greater barriers to be- coming facult y than do men [12], whic h ma y discourage them from seeking facult y jobs at all. Both grant pro- p osal and p eer review success rates can be higher for men than for women, because of implicit biases in the ev alu- ations of the comp etence of women [3, 13]. T ec hnical disciplines, including computer science, often ha ve a nor- mativ e exp ectation of in tellectual brilliance, and in these fields women are less likely than men to seek do ctoral degrees [14]. Experiments using name and gender v aria- tions on resumes ha ve found that both male and female facult y members tend to rate male applicants as more comp eten t, more hireable, and w orthy of more mentor- ing than female applicants [8]. T ak en together, it ap- p ears reasonable to expect strong and p erv asive evidence of gender bias in facult y hiring outcomes across computer science. Other studies hav e argued that the evidence of bias is lacking, even if it ma y hav e existed in the past. F or instance, a review of 30 years of research on the leaky pip eline found that while gender differences were sub- stan tial prior to the 1990s in STEM fields, the gap has since closed [15]. A separate review article survey ed lit- erature on mathematical abilities in children, attitudes to ward math-in tensive fields, and access to, p ersistence in, and rem uneration for facult y , concluding that no evi- dence of systematic gender bias e xists to day [9]. One re- cen t study contro v ersially claimed to find a 2-to-1 prefer- ence for female applicants o ver male applican ts in STEM ten ure-track faculty p ositions, based on a h yp othetical hiring scenario [10]. How ever, the exp erimental design did not include applican t publications, presen tations, or reference letters, and thus it is unclear the degree to whic h these results reflect real preferences, aspirations, or p olitical correctness. Even if the evidence is real, iden- tifying its cause remains difficult. F or instance, some studies argue that the critical v ariable underlying female underrepresen tation is not gender itself but differences in p ersonalit y [16] and structural p osition [17]; better access to resources for hiring, reviewing, and publishing [17– 19]; or the low er likelihoo d of workplace sexual harass- men t [20], that happen to correlate with b eing male. The role of gender in shaping outcomes in facult y hir- ing is difficult to assess, in part, because the hiring pro- cess itself is complicated and opaque. In real facult y searc hes, applicants will v ary along dimensions of gender, pro ductivit y , subfield, doctoral prestige, p ostdo ctoral ex- p erience, references, and more; applican ts apply to man y , but not all searches; and both applican ts and institutions ha ve in ternal, often undeclared preferences. Our aim in this pap er is not to mo del all of these complexities. In- stead, w e adopt the more narro w goal of estimating the effectiv e role of measurable factors like gender, pro ductiv- it y , and institutional prestige on observ ed facult y hiring outcomes. W e do this by form ulating a net work mo del of the yearly matching pro cess of applicants to faculty op enings, which we parameterize to allow us to quan- 1970 1980 1990 2000 2010 Start year 0 20 40 60 80 100 120 140 160 180 Number of faculty hired All hires Female hires FIG. 1. F or the 2659 computer science facult y in our sample (collected in 2011), the distribution of y ears in whic h they w ere first hired as an assistant professor. tify the impact of differen t features of faculty applicants. This approac h allo ws us to inv estigate gender balance in the hiring histories of individual institutions and in indi- vidual faculty placemen t. W e b egin b y describing the facult y hiring and sc holarly pro ductivit y data sets and the statistical features w e de- riv e from them. W e then form ulate a netw ork mo del for facult y placemen t, chec k its accuracy in repro ducing pat- terns found in the real hiring net w ork, and use it to test a v ariety of h ypotheses about the mo del features. Finally , w e discuss our results in the con text of other findings on gender inequality and highligh t strengths and weaknesses of our analysis, before concluding. I I. D A T A AND FEA TURES The primary data set that we used is a comprehensiv e, hand-curated list of the education and academic appoint- men t histories of tenure-trac k or ten ured computer sci- ence faculty [1]. This data set co v ers the 205 departmen- tal or school-level academic units on the Computer Re- searc h Asso ciation’s authoritative F orsythe List of Ph.D.- gran ting departments in computing-related disciplines in the United States and Canada. 3 F or eac h of these units, the data set provides a complete list of regular faculty from the 2011–2012 academic year, and for each of the 5032 faculty listed, it provides partial or complete infor- mation on their education and academic app oin tmen ts, obtained from public online sources, mainly resum´ es and homepages. Within this group, we selected the 2659 faculty who b oth received their Ph.D. from and held their first as- sistan t professorship at one of these institutions, and for whom the year of that hire is kno wn and o ccurred in 3 http://archive.cra.org/reports/forsythe.html 3 1970–2011. Figure 1 sho ws the distribution of these hire dates. The first requirement ensured that we modeled the relativ ely closed North American faculty market; roughly 87% of computing faculty receiv ed their Ph.D. from one of the F orsythe institutions, and past analysis has shown that Canada and the United States are not distinct job mark ets in computer science [1]. A num b er of facult y w ere remov ed in this step b ecause the lo cation of their first assistant professorship was not known; these were mainly senior facult y . The second requiremen t allow ed us to extract a yearly time series of applican ts and op enings, and thus use a more realistic mo del of faculty hiring ov er time. Of the included faculty , women made up 16.1%, whic h was not significantly differen t from the fraction in the discarded set ( p = 0 . 92, χ 2 ), and the changes in insti- tutional rank (see next subsection) were not significantly differen t b et ween men and women in the discarded set ( p = 0 . 325, Mann-Whitney). Thus, our inclusion criteria are unlikely to bias our subsequent results. W e modeled the hiring pro cess using a parametric mo del of edge formation in the faculty hiring netw ork, in which the probabilit y that a particular applican t is matc hed to a particular job op ening dep ends on features of b oth applicant and opening. These features were (i) an applicant’s gender, (ii) the prestige of the hiring in- stitution, (iii) an applicant’s scholarly productivity , (iv) an applicant’s p ostdo ctoral training, (v) the prestige dif- ference betw een do ctoral and hiring institution, and (vi) whether those institutions are in the same or different geographic regions. F or each, we describ e the wa y the feature w as constructed and provide some simple statis- tics describing their relationship to gender. Institutional prestige. F rom the education and ap- p oin tmen t data, we constructed a faculty hiring net work, a directed m ultigraph where each no de is an institution and eac h Ph.D. graduate from an institution u who b e- gan as an assistant professor at v is represented by a single directed edge ( u, v ). Each no de in this net w ork is annotated with its institution’s prestige rank [1], whic h is also given in the primary data set. The prestige rank of an institution quan tifies its abilit y to place its graduates as faculty at other prestigious insti- tutions. F ormally , rank( u ) is the mean rank of u across all orderings that hav e the minimum num b er of “vio- lating” arcs, i.e., an up ward-poin ting arc ( u, v ), where rank( v ) is b etter than rank( u ). Such a ranking is called a minimum violation r anking (MVR) and is a common w ay to measure prestige in so cial systems [21, 22]. The prestige ranking we used was obtained by sampling the MVRs for the full faculty hiring netw ork, and it repre- sen ts a hierarch y on the institutions in whic h only 12% of edges violate the ranking, i.e., only 12% of individ- uals were hired at an institution more prestigious than their do ctorate institution. This ranking correlates with the p opular but widely criticized [23] computer science ranking b y U.S. News & World R ep orts ( r 2 = 0 . 80), but it has the adv an tages of co v ering the complete F orsythe list and b eing based on the collectiv e hiring decisions of the departments themselv es. W e constructed tw o features using these ranks: the rank difference ∆rank( u, v ) b etw een the applicant’s do c- toral institution u and the hiring institution v , and the rank( v ) of the hiring institution alone. Comparing female and male faculty in our sample, w e found no significant difference in the ranks of the doctoral institutions ( p = 0 . 41, Mann–Whitney) or the hiring in- stitutions ( p = 0 . 12, Mann–Whitney). The distribution of the rank differences quan tifies the degree to which ap- plican ts tend to mov e up or down the ranking when they tak e a facult y p osition (see T able I). W e found no signif- ican t difference in the rank differences b etw een men and w omen, b oth including ( p = 0 . 33, Mann–Whitney) and excluding “self-hires” ( p = 0 . 11, Mann–Whitney), i.e., cases in whic h a universit y hires one of its o wn gradu- ates. W e did find a significant difference in the rates of self-hires, with 9.4% of women b eing self-hired com- pared to 6.1% of men ( p = 0 . 02, χ 2 ). Altogether, men and women are trained and hired at similar rates across prestige rankings. do wn up men 1877 (79.3%) 491 (20.7%) w omen 357 (81.0%) 84 (19.0%) T ABLE I. W omen and men mov e up in the prestige rankings at similar rates (excluding self-hires.) Sc holarly pro ductivity . Publication records are an imp ortan t factor in the ev aluation of faculty candidates. F or each applicant we assigned a feature that captures their sc holarly pro ductivity , controlling for subfield v ari- ations, prior to b eing hired in to their first assistant pro- fessorship. T o construct this feature w e first collected a complete publication profile for each facult y from DBLP , an on- line bibliographic database 4 that, in late 2015, indexed o ver 3.1 million publications written by ov er 1.6 million authors, mainly computer scien tists, using man ual name disam biguation as necessary . Through this procedure, w e obtained publication records, including titles and publi- cation dates, for 2528 (95.1%) facult y in our sample. The few individuals for whom we could not iden tify a DBLP profile were assumed to hav e no publications. Publication records in DBLP include journal articles, conference papers (which, in computer science, are p eer review ed), as well as w orkshop pap ers (which often are not). The p erceiv ed v alue of differen t publication t yp es, particular v en ues, or p osition in the author list v aries b y subfield, and w e did not attempt to accoun t for these differences here. Instead, we used the num ber of publi- cations that each faculty had published by one year after starting their assistant professorship, but normalized to con trol for publication rate v ariabilit y across subfields. 4 http://dblp.uni- trier.de/ 4 T o construct this normalization, w e first aggregated the text con tained in all the paper titles of a particular fac- ult y’s DBLP profile, a tec hnique that is common in se- man tic analysis of short texts [24]. W e then applied La- ten t Dirichlet Allo cation [25] to obtain 10 topics or sub- field distributions o ver w ords, which together captured the total v ariation in w ords across all publication records. As a side effect, we also inferred for each facult y a proba- bilit y distribution o ver subfields that characterizes their individual publication record. T o verify that these distri- butions w ere reasonable, w e man ually inspected the most common w ords in each topic and found goo d agreemen t with classic subfields in computer science. Similarly , w e v erified that the inferred topic distributions for a set of w ell-known computer scientists aligned with their known sp ecialities. F or each subfield, we computed a distribution ov er pa- p er coun ts, weigh ted b y each facult y’s inferred emphasis on that subfield. F or each faculty , we computed a sin- gle comp osite z -score for their ov erall pro ductivity by taking a weigh ted av erage of z -scores ov er subfield dis- tributions, with weigh ts giv en by the faculty’s subfield probabilit y distribution. The result is a feature that rep- resen ts eac h p erson’s relativ e pro ductivity , con trolled for their o wn distribution of work across subfields and the norms within those subfields. Pro ductivit y scores do not differ b etw een men and w omen. This is true even when we consider only men and women who mo ved up the ranks and, separately , men and w omen who mo ved down ( p > 0 . 05, Mann–Whitney). Median pro ductivit y scores for men and women in eac h of these categories are rep orted in T able II. W e did find that individuals with postdo ctoral exp erience hav e signif- ican tly higher pro ductivit y scores than individuals with- out p ostdo ctoral exp erience ( p < 0 . 01, Mann–Whitney). This w as true for men and women, separately and to- gether. This is not surprising, as p ostdo ctoral training allo ws more time to write pap ers prior to going on the facult y job market. As w e note b elow, separate treatmen t of pro ductivit y and p ostdoctoral training allow ed us to assess whether or not there is intrinsic v alue in p ostdoc exp erience b eyond providing additional time to publish pap ers. W e note that the pro ductivit y scores of men and w omen do differ when we restrict our analysis to include men and women hired after 2002 (the median start y ear for women). Among these individuals, men are signif- ican tly more pro ductiv e than w omen ( p = 0 . 03, Mann– Whitney). This finding supports the existence of a pro- ductivit y gap in recen t y ears, despite the previously men- tioned studies, which suggest that suc h gaps hav e nar- ro wed or closed ov er time in other disciplines [17, 26]. Geograph y and p ostdo ctoral training. Geogra- ph y and p ostdo ctoral training were captured in t wo bi- nary features. F or the former, w e assigned a v alue of 1 if the pair ( u, v ) spanned t wo institutions in the same geo- graphic region (U.S. Census regions plus Canada), and a 0 otherwise. F or the latter, we assigned a v alue of 1 if a do wn up all men -0.322 -0.207 -0.327 w omen -0.331 -0.215 -0.329 T ABLE I I. Median z -scores by gender and by whether a fac- ult y mov ed up or down the ranking for their faculty posi- tion. W e find no significant differences comparing men and w omen’s pro ductivit y scores in each of these categories. Me- dian v alues are negative indicating that pro ductivity scores are right-sk ewed due to prolific facult y . p erson had any postdo ctoral exp erience recorded in our primary data set, and a 0 otherwise. W e found no difference in the p ercen tages of men and w omen graduating and b eing hired in the same geo- graphic region ( p = 0 . 12, χ 2 ). Of the p eople falling into this category , w e next asked whether mov ement up or do wn in the ranks was linked to gender, and w e found no evidence to suggest that these v ariables were related ( p = 0 . 72, χ 2 ). W e did find, how ever, that for individ- uals who changed geographic regions, men were signif- ican tly more likely than women to hav e mov ed up in rank ( p = 0 . 01, χ 2 ). These results are presented in T a- ble II I. Additionally , conditioned on moving up the ranks, men c hanged geographic regions significan tly more than w omen ( p = 0 . 03, χ 2 ), with 67.8% of men changing re- gions compared to only 48.7% of women. do wn up men 1150 (85.7%) 192 (14.3%) w omen 220 (92.1%) 19 (7.9%) T ABLE I II. F or individuals graduating and b eing hired in separate geographic regions, men are significantly more likely to b e mo ving up the ranks ( p = 0 . 01, χ 2 ). W e found that, in general, w omen were significantly more likely than men to hav e p ostdo ctoral exp erience. 24.1% of women in the dataset completed at least one p ostdoc compared to only 19.3% of the men ( p = 0 . 03, χ 2 ). Ha ving p ostdo ctoral experience, though, did not mak e w omen an y more or less lik ely to mov e up the ranks than men ( p = 0 . 92, χ 2 ), as display ed in T able IV. do wn up men 347 (86.3%) 55 (13.7%) w omen 80 (86.0%) 13 (14.0%) T ABLE IV. F or individuals with p ostdo ctoral experience, men and w omen mov e up the ranks at similar rates ( p = 0 . 92, χ 2 ). Finally , we note that the role of p ostdoctoral exp eri- ence app ears to ha v e c hanged in recen t y ears. Comparing individuals whose first assistan t professorship b egan ei- ther b efore or after 2002, p ostdo ctoral training rates w ere significan tly higher following 2002, 28.1% compared to only 15.5% before 2002 ( p < 0 . 01, χ 2 ). Men and women receiv ed p ostdo ctoral training at similar rates post-2002, 5 29.5% for women and 27.7% for men ( p = 0 . 68, χ 2 ), but the men who did w ere significan tly more productive than the w omen ( p < 0 . 01, Mann–Whitney). W e also note that after 2002 w omen with p ostdo ctoral training w ere not sig- nifican tly more or less pro ductiv e than men without post- do ctoral training ( p = 0 . 44, Mann–Whitney), suggesting that w omen faced additional obstacles which limited their pro ductivit y . I II. A MODEL OF THE F ACUL TY MARKET F acult y hiring is a complicated pro cess, and the par- ticular outcome of a facult y search can dep end on a sur- prising v ariety of factors. Here, w e aim to pare do wn this complexit y to formulate a reasonably simple but still useful mo del of the faculty market as a whole in order to estimate the influence of different features on hiring out- comes in computer science. Our approach uses a data- driv en statistical model of the observ ed outcomes and their features, whic h is distinct from mo dels of strategic in teractions among departmen ts [27]. W e note tw o k ey properties of the facult y mark et: (i) assistan t professor hires are made in rounds, generally once p er year, and (ii) these hires are not indep enden t of eac h other. This second prop ert y comes from the fact that t w o institutions cannot hire the same applican t. A facult y hiring netw ork (where each directed edge ( u, v ) represen ts the hiring a graduate of no de u as an assistan t professor at no de v ) is thus the accumulation of yearly sets of such non-indep enden t hiring edges. W e mo del this net w ork assem bly pro cess by mo deling the ann ual matching of candidates to op enings in eac h y ear of the data. Systematic information on unsuccessful applican ts and unfilled op enings is not generally av ailable for any year, and for this reason we make the simplify- ing assumption that matc hings are made among the ob- serv ed candidates and op enings (the positions that w ere filled) in eac h year. This is not an unreasonable assump- tion: in practice, only a small fraction of facult y op enings go unfilled each year, meaning that the set of successful applican ts is a reasonable appro ximation of the top can- didates across all searches. Th us, for each year t , we first break the observ ed hiring edges { ( u i , v ) } t , where i in- dexes across all candidates, in to tw o “stub” sets, one for the candidates { u i } t and one for the op enings { v } t . W e then generate a matc hing M t on these stubs using a prob- abilistic model f that is parameterized by the pair-level features describ ed in the previous section. Regardless of the reasons why , in practice, hiring com- mittees prefer applicants trained at more prestigious de- partmen ts ab out 80% of the time [1]. W e mo del this and other preferences of a typical hiring committee via a logistic function for the pairwise probabilistic mo del: f ( ~ x [ u i , v ] , ~ w ) ∝  1 + e − ~ x [ u i ,v ] · ~ w  − 1 , (1) where ~ x [ u i , v ] is a vector of features of the candidate- op ening pair u i , v , and ~ w is the global set of weigh ts on those features that w e learn from the data. This c hoice of f allows us to automatically capture t w o imp ortan t sp ecial cases: if f is indep endent of ~ x , then rank and other features play no role and the matching is equiv alent to the p opular configuration random graph mo del [28]; when f is a step function on rank, and in- dep enden t of other features, then hires are chosen uni- formly at random from those trained at more prestigious departmen ts, which is equiv alent to the MVR ranking metho d used in [1]. The step function is the simplest f that depends on some of our features, and w e use it as a baseline mo del later in order quantify the impro vemen t from incorp orating additional mo del features. Applican ts may also prefer op enings at highly rank ed departmen ts, desiring the prestige and resources asso ci- ated with these institutions. W e mo del this preference b y filling the op enings { v } t sequen tially , choosing an unfilled op ening to fill with probability prop ortional to 1 / rank( v ) (where more highly ranked departmen ts ha v e smaller rank scores). Through this sequential matc hing pro cess, our mo del fills each op ening in a given year t from the av ailable candidates in that y ear. Applying this pro cess for each y ear t from 1970 to 2011, the model as- sem bles a full faculty hiring netw ork. It is worth noting that this mo del is lo osely similar to the p opular exp o- nen tial random graph mo del [29]; ho wev er, in our formu- lation, edge formation is ordered and not indep enden t, whic h requires a sligh tly different treatmen t. W e score the qualit y of our model by measuring its to- tal error with resp ect to the observ ed placements, where total error is defined as the mean squared error (MSE) in the placemen ts plus an L1 regularization term to preven t the mo del from ov erfitting. Mathematically , err = 1 m m X i =1 [observ ed( u i ) − mo del( u i )] 2 + λ X k | ~ w k | , (2) where observed( u i ) is the observed placemen t rank of candidate i and mo del( u i ) is the sim ulated placement rank. Using the MSE allo ws the mo del to receiv e par- tial credit for matc hing an applican t to an opening with rank similar to the observed rank, rather than, for ex- ample, receiving credit only if the applicant matc hes to the observed op ening (which simply counts the num b er of correct placemen ts). T o estimate the mo del’s param- eters ~ w , we use a standard implementation of a direct searc h optimization algorithm (Nelder-Mead). A. Mo del chec king As a first step, we c heck that syn thetic faculty hiring net works produced b y our mo del hav e similar structural patterns to the observed netw ork. W e do this for each of three choices of f , the logistic function of Eq. (1) using all six features, as well as its t w o sp ecial cases, a uniform 6 mo del f observ ed uniform step logistic mean geo desic path length 2.23 2.05 ± 0.01 2.07 ± 0.01 2.16 ± 0.01 mean lo cal clustering co efficient 0.25 0.34 ± 0.01 0.38 ± 0.01 0.22 ± 0.01 % recipro cated hires 18.95 14.52 ± 0.81 4.17 ± 0.34 13.93 ± 0.69 % recipro cating institutions 14.25 13.23 ± 0.77 1.72 ± 0.21 9.86 ± 0.61 % self-hires 6.62 0.93 ± 0.18 3.74 ± 0.27 1.95 ± 0.25 % placements within same region 40.54 21.27 ± 0.77 24.48 ± 0.76 29.15 ± 0.75 T ABLE V. Net work summary statistics used in model chec king of uniform, step, and logistic choices of f . In each ro w, b oldface indicates the mo del that best repro duces that characteristic of the observ ed netw ork. function and a step function. Using standard netw ork summary statistics [30], such as the mean geo desic path length and the mean local clustering co efficient, as w ell as hiring-sp ecific statistics on recipro cal hiring, self-hiring, and within-region placement, we compare the observed and simulated net works. T able V summarizes the results of this exercise. In general, w e find very go o d agreement b et ween the statistical prop erties of the real netw ork and those gen- erated by each of our models, with the logistic mo del p erforming b est o verall. Eac h of our mo dels underesti- mates the rates of recipro cal hiring and self-hiring. This suggests that additional factors not presen t in our mo del lik ely influence these types of hires, p erhaps related to the pre-existing so cial and professional connections asso- ciated with such hires. Finally , we v erify that the feature weigh ts learned by our model are consistent under cross-v alidation in whic h sets of fiv e randomly selected years of data are set aside for testing. F eature w eights are largely stable across runs with only minor fluctuations that do not hav e a signifi- can t impact on modeling error. IV. RESUL TS In the following sections, w e examine gender’s role in univ ersity faculty hiring at three levels by inv estigating (i) system-wide effects, (ii) hiring results for individual institutions, and (iii) hiring results for individual candi- dates. W e conclude b y forecasting when computer sci- ence will reach gender parit y , should w omen’s presence in the field con tin ue to grow at the current rate. A. Mark et-level analysis W e trained a series of placement mo dels b y incorp o- rating, one at a time, the attributes describ ed in the pre- vious section. The order in which attributes were added to the mo del was determined greedily: each remaining attribute was added separately to the previous mo del, and the attribute pro ducing the greatest reduction of er- ror was built into the subsequen t mo del. Gender w as incorp orated last in order to determine if it significan tly impro ved mo deling results b eyond the effects of all other Rank difference + Productivity + Rank of hiring inst. + Postdoc experience + Geography + Gender -10% -15% -20% -25% -30% -35% -40% -45% Error reduction relative to baseline -25.4% * -27.4% * -28.0% -27.9% * -28.7% -28.5% FIG. 2. Reduction of mo deling error as features are added to the model. Percen t reductions are computed relative to the step function mo del as a baseline. Median p ercent reduc- tions are reported for each mo del, and attributes pro ducing a significant reduction in error ( p < 0 . 05, Mann–Whitney) are mark ed with braces and asterisks. v ariables. Figure 2 shows the exten t to which mo deling error decreased as attributes were incremen tally incorp o- rated. The list of attributes added to the mo del, in decreasing order of error reduction, was (i) rank difference b et w een do ctoral and hiring institutions, (ii) scholarly pro ductiv- it y , (iii) rank of hiring institution, (iv) p ostdoctoral train- ing, and (v) whether do ctoral and hiring institutions were in the same geographic region. It is p erhaps unsurpris- ing that rank difference and pro ductivity yield the largest impro vemen ts in mo deling results as these attributes are kno wn to play key roles in faculty hiring. Incorp orat- ing the rank of the hiring institution also significantly impro ves mo deling results ( p < 0 . 01, Mann–Whitney). Based on the sign of the inferred co efficien t, this suggests that the most prestigious univ ersities are more selectiv e in their hires and p oten tially v alue prestige more than lo wer-rank ed univ ersities. Neither p ostdoctoral exp erience nor geographic infor- 7 mation alone produced a significant change in mo deling error. T ogether, how ever, these features accounted for a small but significant improv emen t. Because the pro- ductivit y score had already b een greedily added to the mo del prior to p ostdo ctoral training, this result implies that postdo ctoral training, in general, is only nominally useful beyond the exten t to which it offers a trainee ad- ditional time to publish more pap ers and to thereb y in- crease his or her pro ductivity score. Geographic infor- mation, similarly , has little effect on mo deling error. On its own, this finding suggests that issues of mobility do not strongly and systematically affect the placement of all faculty . W e noted in Sec. 2, how ev er, that men who mo ved up in the ranks are more likely than w omen who mo ved up to hav e changed geographic regions. T ogether, these findings suggest that mobility may pla y a small but real role in placement differences for some groups of men and women. Finally , the addition of gender into the placemen t mo del did not significantly improv e mo deling results. W e found this to b e true both when we computed placement error for all faculty , and for women, separately . That the incorp oration of gender do es not significantly impro v e global error suggests that gender in and of itself do es not systematically affect all hires b eyond p otential indirect effects enco ded in other features, suc h as pro ductivity . This finding echoes historical work [31], which suggests that gender discrimination within science is not evenly distributed and warns that ignoring this non-uniformity risks promoting inequality . That being said the w eigh t assigned to gender was nev ertheless non-zero, indicating that a subtle difference do es exist. T o conv ert this difference into more tangi- ble terms, w e calculated the num b er of additional pa- p ers a female candidate w ould need to publish in or- der to achiev e the same job placement as an otherwise equiv alent male candidate. Across subfields, on a ver- age, women must publish approximately one additional pap er—a roughly 10% increase in pro ductivit y—in order to comp ete on even footing with men. B. Institution-lev el analysis F or facult y hiring to b e free of uniform and system- atic gender bias do es not suggest that inequalit y cannot exist at the level of individual institutions. In this sec- tion, we explore this possibility directly b y comparing the observed hiring at each institution with the distri- bution of outcomes dra wn from our generative mo del of facult y placemen t. Using all features listed in previous sections, we simulated 1000 complete hiring histories, re- quiring as before that universities comp ete for candidates during eac h year of the process. F or eac h simulation, w e trac ked the n umber of male and female hires by year and b y institution, resulting in an ev aluation of the gen- der balance of eac h department, taking into account the n umber of women on the job mark et when the depart- men t was hiring and the likelihoo d that those candidates w ould hav e b een hired by the institution. The result is a set of institution-sp ecific assessments that accommo- date the non-independence of hires while con trolling for placemen t likelihoo ds of candidates. In comparing each institution’s actual num b er of fe- male hires to the exp ected n umber under simulation, we find that most institutions p erform very closely to their exp ected v alues. There are, how ev er, institutions that exceed or fall short of the mo del’s exp ectations. Figure 3 highligh ts universities in each of these three categories. By comparing the results of many institutions, w e ask ed whether female hiring patterns change as a func- tion of rank. Figure 4 illustrates the difference b et ween actual and expected coun ts of w omen at the top 50 uni- v ersities, sorted by rank. W e note that top-ranked in- stitutions (ranks 1–10) tend to hire more w omen than exp ected, while slightly low er-ranked institutions (ranks 11–20) typically hire fewer. This pattern ma y suggest that efforts made by top institutions to rectify instances of gender imbalance in their own departments could come at the expense of imp eding similar efforts b y low er-ranked institutions. C. Candidate-lev el analysis Ha ving analyzed facult y hiring at the system level and at the level of individual institutions in previous sections, w e now in vestigate the placemen t of individual faculty . The complete sim ulations of the faculty market used in the institution-lev el analyses were re-analyzed for each individual faculty . Sp ecifically , for each individual, w e compiled a list of sim ulated placemen ts and their frequen- cies, constituting a distribution of plausible outcomes for that p erson. By comparing the ranks of the insti- tutions in an individual’s list of plausible outcomes to that of their hiring institution, we obtained a distribu- tion representing the amoun ts by which eac h p erson has o ver- or under-performed relativ e to their simulated out- comes. W e separated these individuals by gender, and found that men and women meet or exceed mo del exp ec- tations at similar rates, though women are more likely to exceed exp ectations ( p < 0 . 01, Mann–Whitney). F or under-p erforming individuals, how ev er, men tend to fall short of their exp ectations b y significantly larger amoun ts ( p < 0 . 01, Mann–Whitney). W e also find that individuals with p ostdoctoral train- ing are more lik ely to outp erform model exp ectations than those without this exp erience ( p < 0 . 01, χ 2 ). This result is true for men and women, b oth separately and together, although w omen tend to exceed their exp ec- tations b y larger amounts ( p < 0 . 01, Mann–Whitney). This implies that in the past, postdo ctoral exp erience ma y hav e provided a strategic adv an tage to women lo ok- ing to mov e up the ranks of the prestige rankings. With more men receiving p ostdo ctoral training in recen t y ears, ho wev er, it app ears that what was once a comp etitive 8 '70 '80 '90 '00 '10 0 2 4 6 8 Cumulative female hires UC Berkeley Simulated Actual '70 '80 '90 '00 '10 Princeton University '70 '80 '90 '00 '10 0 2 4 6 8 Brigham Young University 0 2 4 6 8 0 2 4 6 8 Distribution of final counts FIG. 3. Three examples of mo del-based sampling of universit y-sp ecific female hire distributions. Eac h green tra jectory denotes the cumulativ e num b er of hires for a single simulation of the placement model at the indicated univ ersity . Running many sim ulations creates the distribution ov er final counts, sho wn on the right. The actual tra jectory of hires made by the institution (within the data set) and the resulting final coun t are highligh ted in black. UC Berkeley , Princeton, and Brigham Y oung represen t examples of exp ected, female-skew ed, and male-skew ed hiring, as indicated by the lo cation of the actual v alue within eac h sampled distribution. FIG. 4. Comparison of actual and expected female hiring o ver the top 50 institutions. Dots represent actual v alues minus exp ected v alues calculated from distributions samples as in Fig. 3. The shaded region denotes the 25th-75th p ercen tiles, based on mo deling outcomes. Six particular univ ersities are annotated. T op 10 schools hire slightly ab o ve exp ectations while ranks 11–20 hire below expectations. This suggests that the efforts by the highly-ranked schools to rectify any gen- der imbalance ma y hav e imp eded the efforts of lo w er-ranked sc ho ols hoping to do the same. strategy may no w b e the norm. Grouping individuals together by hiring year, we in- v estigated ho w placement error is distributed o ver time. This allo ws us to assess the degree to which faculty hir- ing app ears to hav e changed ov er the timeframe spanned b y the dataset. L ik e the previous analysis, this is equiv- alen t to looking at the av erage amount by which men and w omen ov er- or under-p erform, collectively , in each hiring year. F or instance, a pattern of women tending to under-p erform early in the time p eriod, and to ov er- p erform later in the time perio d w ould be consisten t with impro ved conditions for female facult y to day . Instead, w e see noisy , but relativ ely flat functions for the placement errors for both women and men (Fig. 5), with the differ- 1975 1980 1985 1990 1995 2000 2005 2010 Hiring year 0.3 0.2 0.1 0.0 0.1 0.2 0.3 Mean placement error Women Men FIG. 5. Mean placement error by y ear. Placemen t error is computed as the difference b etw een the rank of the institu- tion where the p erson was hired and the rank of the insti- tution where they placed under sim ulation. Higher v ariance in female placement error is within fluctuations exp ected due to low er female representation in the data set. Adjusted for y early represen tation in the data, error is neither systemati- cally increasing nor decreasing in time. ence in fluctuations by gender attributable to the differ- ence in sample size. This pattern indicates that mo del errors in either direction are equally likely for men and for w omen, and for b oth recen t hires and hires from several decades ago. D. Long-term forecast for gender parit y Ov er the four decades spanned b y our data, the pro- p ortions of received do ctoral degrees and assistant pro- fessor p ositions held in computer science by women hav e b oth steadily increased, from around 5% to roughly 20% (Fig. 6). Ho wev er, the share of new facult y positions held b y w omen is on av erage about 1% lo w er than the share of do ctorates, whic h reflects the well-documented leakiness 9 of the academic training pip eline [7]. While not a large n umber in magnitude, a 1% gap is a substan tial prop or- tional difference (ab out 7–20%) giv en that the gender ratio is so hea vily sk ewed to w ard men. Nev ertheless, the long-term trend in computer science is tow ard gender parity . T o estimate when women and men will hold equal shares of new faculty p ositions, we fitted a simple linear mo del to the historical trend and extrap olated it into the future (Fig. 7). Under this mo del, the share of p ositions held by women increases by 0.43% p er y ear on av erage, meaning that it will take roughly 60 y ears from 2012 to reac h parity at the assistant professor lev el, with a 95% confidence interv al of 30–100 years. F ull gender parity across all lev els of faculty should then o ccur 30–40 years later, when the first gender-parity cohort of assistan t professors begins to retire. V. DISCUSSION Here, we used a unique data set on the hiring of as- sistan t professors in computer science from 1970–2011 to measure the imp ortance of six features of candidates on observ ed hiring outcomes. Among these, do ctoral prestige and scholarly pro ductivit y play an outsized role, while gender alone do es not app ear to b e a significant factor in the typical hiring decision. At face v alue, these findings are consistent with a system that is not ov ertly biased by a candidate’s gender. Ho wev er, we also found evidence of (i) unexp ectedly gender im balanced hiring patterns at individual institu- tions, (ii) significant differences b etw een genders in rates and the effects of publishing and p ostdoctoral training, (iii) differences b et ween men and women who mov e up the prestige ranking, and (iv) evidence of that higher rank ed institutions’ success at hiring female faculty ma y 1970 1980 1990 2000 2010 Start year 0.00 0.05 0.10 0.15 0.20 Fraction that are women Faculty PhDs (NCES) FIG. 6. Time series of the fraction of assistant professor hires since 1970 in our dataset that are w omen (green; with 95% confidence in terv als around the mean), and the fraction of computer science do ctoral recipients since 1970 that are w omen (blac k). 1975 2000 2025 2050 2075 2100 2125 0.00 0.25 0.50 0.75 1.00 Fraction of hired faculty Women Men FIG. 7. Gender ratio of assistant professors in computer sci- ence, by gender, and a pro jection for when gender parity will b e reac hed. If the historical trend con tin ues unaltered, gender parit y will o ccur in appro ximately 2075. Shaded regions rep- resen t extrapolated 95% confidence in terv als from an ordinary least squares regression. b e limiting similar efforts at marginally less highly rank ed institutions. The apparen t conflict b et w een these t wo sets of findings about the same facult y mark et sho ws that the role of gender in faculty hiring is subtle and generally not well characterized by simple statistics or broad gen- eralizations. Ov erall, our results suggest that the actual facult y hiring mark et in computer science is neither ex- tremely dire for w omen [8] nor extremely fa v orable [10]. Under our mo del, the inclusion of candidate gender did not significantly impro ve its ability to correctly place facult y ov erall. There are at least three plausible inter- pretations of this b ehavior. First, gender could b e an irrelev ant feature in faculty hiring. This interpretation is implausible b ecause w e also found that gender correlates with postdo ctoral training, pro ductivit y , and geographic mobilit y , esp ecially in the past 10 y ears. Second, the ef- fect of gender ma y not b e included realistically in the mo del. Evidently , a uniform p enalt y or adv antage based on gender do es not help reduce placement error rates, and so the gender feature received a weigh t near zero. Or third, the primary effects of gender on placemen t are already incorp orated into the mo del through other fea- tures that correlate with gender. This latter interpretation is particularly plausible. F or assistan t professors who started since 2002, productivity scores correlate with gender, with men b eing on a v erage more pro ductive than women with the same amount of training ( p < 0 . 01, Mann-Whitney). Moreo v er, the pro- ductivit y of women with p ostdoctoral training is not sig- nifican tly different from men without it ( p = 0 . 44, Mann- Whitney), and under our mo del, w omen need to be ab out 10% more pro ductive, on av erage, in order to place at equal rates as men. That is, productivity already en- co des gender-based differences, making a separate gen- der v ariable in the mo del redundan t. The origin of this pro ductivit y gap seems unlikely to b e related to inheren t 10 differences in talent or effort, and may instead b e related to differential access to resources and men toring [19], greater rates of hostile work environmen ts or sexual ha- rassmen t [20], differences in self-p erceptions [32], or other gender-correlated factors. Additional researc h is needed to inv estigate these p ossibilities. Our findings that supp ort the existence of a gender- driv en pro ductivity gap in recent y ears are at odds with sev eral studies indicating that such gaps ha v e narrow ed o ver time or perhaps closed altogether in other disciplines [17, 26]. These studies, how ever, examine the total num- b er of publications and citations accumulated ov er one’s en tire career whereas w e fo cus on an individual’s publi- cation record up until one year after b eing hired. Dif- ferences in pro ductivit y at this stage hav e b een noted previously [33] and are most relev ant to our study of fac- ult y hiring, as these differences lik ely influence hiring as w ell as tenure decisions and th us the individuals observed in our dataset. Indeed, w e find that w omen are ov errep- resen ted on the low end of our productivity measure and publish few er pap ers p er y ear on a verage for the first sev eral y ears of employmen t. A better understanding of the causes b ehind this lag in productivity w ould inform facult y ev aluation pro cedures and tenure p olicies, p oten- tially impro ving retention of women at this career stage. The pro ductivit y gap also suggests that p ostdoctoral training has been one w ay for women to comp ete on an equal basis with men in the faculty market. F or faculty who started prior to 2002, the rate of p ostdo ctoral train- ing was indeed higher among women than men, whic h ma y reflect a comp ensatory adaptation to a biased sys- tem [34]. Since 2002, how ev er, these rates hav e equal- ized, meaning that in a typical faculty searc h to day , men are lik ely to app ear more pro ductive, on av erage, than w omen. Institutional self-hiring, i.e., b ecoming faculty at one’s doctoral institution, may reflect a separate kind of compensatory adaptation. Across 40 y ears, w omen ha ve b een hired b y their doctoral institutions at a greater rate than men, and this difference has grown significantly since 2002. Determining the extent to which these pat- terns reflect strategic resp onses to a changing market w ould shed new light on the underlying market struc- ture. The long-term trend in the gender ratio in computer science faculty hiring is to ward parity . The pace, how- ev er, is glacial, and w e estimate that it will take roughly 60 y ears to reach. There are tw o main reasons to w ant to accelerate this trend: (i) social justice and the provision of equal opp ortunities [35, 36], and (ii) increased scien- tific inno v ation, creativit y , and pro ductivity [5, 6, 37, 38]. Ac hieving parit y so oner, how ev er, is lik ely to require no vel and concerted efforts, as the faculty gender ra- tio correlates strongly with the do ctoral gender ratio (Fig. 6), suggesting that relatively little has c hanged, fun- damen tally , ov er the past 40 years. F or an individual computer science departmen t aim- ing to improv e its faculty gender balance, the non- indep endence of hires p oses a thorn y problem. W e ob- serv e a rank-dep enden t pattern indicating that more highly rank ed departments tend to ha ve b etter than exp ected rates of female facult y hiring and reten tion (Fig. 4), p oten tially at the expense of those departments rank ed just b elo w, e.g., ranks 1–10 vs. 11–19, and ranks 20–25 vs. 26–40. Even if all departmen ts wished to hire more female faculty , the more highly rank ed institutions will tend to hav e a comp etitiv e adv antage in attracting any candidates. Thus, if man y departmen ts are com- p eting to hire a small num b er of female candidates, the lo wer-rank ed departments will tend to lose out. Broad- ening the p o ol of female candidates is one solution to this problem, whic h a recent exp erimen tal study show ed has a direct improv emen t on the gender ratio among faculty hires [39]. Because the hiring netw ork data set is a snapshot of regular faculty in the United States and Canada in the 2011–2012 academic year, it necessarily omits an y infor- mation about faculty who left or retired from computer science prior to 2012, who were hired since 2012, or who w ere hired at the associate or full professor level during our study p eriod, e.g., faculty who sp en t time in indus- try or who did their assistan t professorship outside of computer science or outside the U.S. and Canada. As a result, hiring and retention are confounded in our anal- ysis, and the current gender imbalance at some depart- men ts ma y b e smaller than what w e estimate. W ere in- formation on these missing individuals to b ecome av ail- able, our model could b e used to study questions ab out the leaky pip eline, e.g., do certain institutions or groups of institutions contribute more or less to w omen lea ving the pip eline, or to compare the dynamics of the new-hire mark et and the senior-hire market. Another limitation of this data set is that it do es not include information on other facult y v ariables, such as their ethnicity , whic h can be particularly sk ewed, e.g., with African American facult y [40], so cio-economic bac kground, or nationality . These represent imp ortan t directions for future research. The productivity feature dev eloped here could p oten- tially b e impro ved. F or simplicit y , w e assigned all publi- cations equal weigh t in our analysis, whic h fa vors quan- tit y ov er quality . A b etter feature, ho w ever, w ould com- bine a candidate’s scholarly record with an estimate of its sc holarly quality and the author’s level of contribution. Ho wev er, such an extension w ould b e highly non-trivial, in part b ecause qualit y is difficult to measure accurately and automatically , across subfields. In fact, reliably as- sessing publication quality is hard even for humans, par- ticularly when that contribution is interdisciplinary [41]. An automated to ol for doing so would hav e v alue b oth for the scientometrics and text mining communities as w ell as hiring committees. In our model, we used a logistic function to score p o- ten tial matc hings b etw een candidates and hiring insti- tutions. Allo wing this function to tak e a more complex form could impro ve the mo del’s accuracy , either through the incorp oration of interaction terms or by adopting a ric her functional form in place of Eq. (1). Though we 11 do not explore these possibilities here, suc h modifications could enrich future analyses in this area and offer a source of flexibilit y for adapting our mo deling framework to suit other applications. F acult y hiring netw orks provide a p ow erful new to ol for understanding the dynamics of academic disciplines, and for in vestigating the role of different factors in shap- ing academic careers. The computer science hiring net- w ork rev eals substantial evidence that gender inequalit y is present, subtle, and non-uniform. F or predicting fac- ult y placement, do ctoral prestige and relative pro ductiv- it y appear to b e the most imp ortan t v ariables. How ev er, the correlation b et ween pro ductivit y and gender raises the questions of wh y , ho w the gap can b e closed, and ho w our assessments can b e informed by its underlying causes. Although the details are different, the computing industry has an equally large gender imbalance. Employ- ing a similar approach to industrial hiring netw orks and pro ductivit y ma y shed new light on its underlying causes and the means to address it. VI. A CKNOWLEDGMENTS The authors thank Bailey F osdic k, Abigail Z. Jacobs, Win ter Mason, and Jennifer Neville for helpful con ver- sations, and the BioF rontiers Institute at the Universit y of Colorado Boulder for computational resources. This w ork was supp orted in part by the Butcher F oundation. [1] A. Clauset, S. Arb esman, and D. B. Larremore, Science Adv ances 1 , e1400005 (2015). [2] N. Ellemers, P olicy Insights Behav. Brain Sci. 1 , 1 (2014). [3] A. Kaatz, B. Gutierrez, and M. Carnes, T rends Phar- macol. Sci. 35 , 371 (2014). [4] H. Shen, Nature 495 , 22 (2013). [5] S. E. Page, The Differ enc e: How the Power of Diver- sity Cr e ates Better Groups, Firms, Scho ols, and So cieties (Princeton Universit y Press, 2008). [6] W. Kets and A. Sandroni, “Challenging conformity: A case for diversit y ,” (2015), MPRA P ap er No. 68166. [7] C. Hill, C. Corbett, and A. St Rose, Why so few? Women in Scienc e, T e chnolo gy, Engineering, and Math- ematics. (ERIC, 2010). [8] C. A. Moss-Racusin, J. F. Dovidio, V. L. Brescoll, M. J. Graham, and J. Handelsman, Pro c. Natl. Acad. Sci. USA 109 , 16474 (2012). [9] S. J. Ceci, D. K. Gin ther, S. Kahn, and W. M. Williams, Psyc h. Sci. Public In terest 15 , 75 (2014). [10] W. M. Williams and S. J. Ceci, Pro c. Natl. Acad. Sci. USA 112 , 5360 (2015). [11] J. M. Sheltzer and J. C. Smith, Pro c. Natl. Acad. Sci. USA 111 , 10107 (2014). [12] S. M. V an Anders, Sex Roles 51 , 511 (2004). [13] R. v an der Lee and N. Ellemers, Pro c. Natl. Acad. Sci. USA 112 , 12349 (2015). [14] S.-J. Leslie, A. Cimpian, M. Meyer, and E. F reeland, Science 347 , 262 (2015). [15] D. I. Miller and J. W ai, F rontiers in Psych. 6 , 37 (2015). [16] S. J. Correll, American Journal of So ciology 106 , 1691 (2001). [17] Y. Xie and K. A. Shauman, Am. Sociol. Rev. 63 , 847 (1998). [18] J. R. Cole and H. Zuc kerman, Adv ances in Motiv ation and Achiev ement , 217 (1984). [19] S. J. Ceci and W. M. Williams, Pro c. Natl. Acad. Sci. USA 108 , 3157 (2011). [20] R. Ilies, N. Hauserman, S. Sch wochau, and J. Stibal, P ersonnel Psyc hology 56 , 607 (2003). [21] H. de V ries, Animal Behavior 55 , 827 (1998). [22] J. Henrich and F. J. Gil-White, Ev olution and Human Beha vior 22 , 165 (2001). [23] M. N. Bastedo and N. A. Bo wman, American Journal of Education 116 , 163 (2010). [24] L. Hong and B. D. Davison, in Pr o c. First Workshop on So cial Me dia Analytics (ACM, 2010) pp. 80–88. [25] D. M. Blei, A. Y. Ng, and M. I. Jordan, The Journal of Mac hine Learning Research 3 , 993 (2003). [26] P . v an Arensb ergen, I. v an der W eijden, and P . v an den Besselaar, Scientometrics 93 , 857 (2012). [27] J. Klein b erg and S. Oren, in Pr o c. 2015 Conf. on Innova- tions in The or etical Comp. Sci. (ACM, 2015) pp. 51–60. [28] M. Molloy and B. A. Reed, Random Structures and Al- gorithms 6 , 161 (1995). [29] G. Robins, P . Pattison, Y. Kalish, and D. Lusher, So cial Net works 29 , 173 (2007). [30] M. E. J. Newman, Networks: An Intr o duction (Oxford Univ ersity Press, Oxford, 2010). [31] J. R. Cole, F air scienc e: Women in the scientific c om- munity (New Y ork: Columbia Universit y Press, 1987). [32] B. Chang, Disp el ling Computer Scienc e Gender Myths: Per ceive d Confidenc e L evels Betwe en Under gr aduate Men and Women , B.S. thesis, Carnegie Mellon Universit y (2005). [33] J. S. Long, Social F orces 71 , 159 (1992). [34] K. R. Wyche, Go od Is Not Enough: A nd Other Unwrit- ten Rules for Minority Pr ofessionals (Penguin Publishing Group, 2009). [35] J. F. Dovidio and S. L. Gaertner, Journal of Social Issues 52 , 51 (1996). [36] F. J. Crosb y , A. Iyer, S. Cla yton, and R. A. Do wning, American Psychologist 58 , 93 (2003). [37] N. Bassett-Jones, Creativity and Innov ation Manage- men t 14 , 169 (2005). [38] Catalyst Information Center, “Why diversit y matters,” (2013), h ttp://bit.ly/1OEJjFU. Accessed 22 Nov em b er 2015. [39] J. L. Smith, I. M. Handley , A. V. Zale, S. Rushing, and M. A. Potvin, BioScience 65 , 1084 (2015). [40] W. R. Allen, E. G. Epps, E. A. Guillory , S. A. Suh, and M. Bonous-Hammarth, Journal of Negro Education 69 , 112 (2000). [41] L. R. Lattuca, Cr e ating Inter disciplinarity: Inter disci- plinary R ese ar ch and T e aching Among Col le ge and Uni- versity F aculty (V anderbilt Universit y Press, 2001).