Multivariate Meta-Analysis: Contributions of Ingram Olkin

Statistic al Scienc e 2007, V ol. 22, No. 3, 401– 406 DOI: 10.1214 /07-STS239 c  Institute of Mathematical Statisti cs , 2007 Multiva riate Meta-Analysis: Contribu tions of Ingram Olkin Betsy Jane Becker Abstr act. The r esearc h on meta-analysis an d particularly m ultiv ariate meta-analysis has b een greatly influen ced by the wo rk of Ingram Olkin. This pap er do cuments Olkin’s con tribu tions b y wa y of citation count s and outlines seve ral areas of con tribu tion b y Olkin and his academic descendan ts. An academic family tree is pro vided . Key wor ds and phr ases: Meta-a nalysis, multiv aria te. 0. INTRODUCTION Muc h of th e researc h on statistic al metho ds for meta-analysis in the last thr ee decades has b een in- fluenced by In gram Olkin, either through his direct con tributions or through the w ork of h is studen ts and th eir academic descendants. W e indicate the ex- ten t of this influence and presen t a tree of Olkin’s academic d escendan ts wh o hav e made, or are mak- ing, con tributions to researc h in meta-analysis. W e then consider the outcome metrics that h a v e b een used in the m ultiv ariate meta-analysis conte xt and briefly review key r esu lts for eac h metric, thus show- ing O lkin’s seminal influence on this imp ortan t sub- field of meta-analysis. 1. OLKIN’S INFLUENCE ON MET A-ANAL YSIS Meta-a nalysis is a set of metho ds for combining and analyzing r esults from s er ies of related stud- ies. Glass ( 1976 ) coined the term “meta-a nalysis,” but the idea of summarizing study results is m uc h older, with referen ces dating to the turn of the last cen tury (e.g., Pea rson, 1904 ). Muc h of the litera- ture on metho ds for meta-analysis deals with the Betsy Jane Be cker is Pr ofessor of Me asur ement and Statistics, Col le ge of Educ ation, Florida S tate University, T al lahasse c, Florida 32306, USA e- mail: bb e cker@fsu.e du This is an elec tr onic re pr int of the o riginal ar ticle published by the I ns titute of Mathematical Statistics in Statistic al Scienc e , 200 7 , V ol. 22, No. 3, 401– 4 06 . This reprint differs from the original in pagina tio n a nd t yp ogr aphic detail. univ ariate case—one end p oint p er study . Such end - p oints can b e represent ed by correlations, mean dif- ferences, prop ortions, o dds ratios (or log o d ds) and ev en observ ed probabilities. The first of O lkin’s cont ribu tions to meta-analysis (Hedges and Olkin, 1980 ) examined the in tuitiv ely app ealing v ote-coun ting metho ds used in many re- searc h syn theses and traditional literature reviews. V ote coun ting enta ils counting the n umber of studies that h a ve statistically significan t resu lts in sup p ort of, and counte r to, a particular h yp othesis, as w ell as those with n on s ignifican t results. The cat egory with the most vot es (or more than some sp ecific prop ortion of v otes) “wins” and the set of all re- sults is then c h aracterized as supp orting that view (e.g., if half of the stud ies ha v e signifi can t tests in fa v or of a h yp othesis, the s tudies are viewed as su p- p orting the hypothesis). Hedges and O lkin sho we d that the statistica l p rop erties of th is appr oac h we re problematic—in that more evidence can lea d to p o orer decisions. Since t hen Olkin has authored or c o-authored 39 more articles or b o ok c hapters and one b o ok on meta-analysis. The influ ence of his w ork is sho wn b y the fact that these do cuments ha v e generated o ve r 5600 citatio ns. (Based on searc h es of the W eb of Science at http://8 0isi4.is iknowledge.com.proxy.lib.fsu.edu/ using the author names “Olkin I*,” “Hedges L*,” “Gleser L*” and “Samp son A*.”) His b o ok Statis- tic al Metho ds for Meta-analysis with Larry Hedges (Hedges and O lkin, 1985 ) is something of a cita- tion classic, having b een cited at least 3270 times. Ho wev er, O lkin’s articles and b o ok c h apters are also 1 2 B. J. BECKER highly cited, w ith the num b er of citations p er work ranging from 0 to 916 with a mean coun t of 62.5 cita- tions ( S D = 157 . 4) and a m edian count of 20.5 cita- tions p er article. [As is t ypical of citati on counts, the distribution of citation coun ts p er article is h ighly sk ew ed (sk ewness co efficient = 4.8), su ggesting that the median citation count p er article is th e more appropriate measure of central tendency .] Th e ma- jorit y of this work is collab orativ e—30 of th ese pa- p ers are co-authored, with the mean num b er of co- authors across all 39 d o cuments b eing 2.74 ( S D = 3 . 5). As might b e exp ected from the recipient of the Elizab eth L. S cott Aw ard from the Committee of President s of S tatistica l So cieties (in 1998 ), o v er half (17) of Olkin’s 30 co-a uthored w orks were wr itten with at least one female co-author. 2. CONTRIBUTIONS TO MET A-ANAL YSIS OF OLKIN’S A CADEMIC DESCEND ANTS Besides Olkin’s o wn con tributions to meta-a nalysis, individuals that he has men tored and trained hav e also made many contributions to this literature— some wr iting dissertations on meta-analysis topics. (Ap ologies are made to an y studen ts of Olkin and his descendan ts who ha v e inadve rtently b een omitte d from this analysis.) All first-generation descendants w ere student s at Stanford Unive rsity , though n ot all earned degrees in the Departmen t of Statistics. In addition, student s of those students are considered, and so on, through seve ral generations of Olkin aca- demic “descendants.” These individu als are displa yed in Figure 1, the Olkin meta-analytic family tree. The ye ars shown in the figur e are the graduation dates for eac h p erson; dissertations concerning meta-analysis metho ds are included in th e r eferen ce list as well. Th e tree sho ws Fig. 1. The Olkin m eta-analytic fami ly tr e e. MUL TIV ARIA TE MET A-A NAL YSIS 3 on the b ottom-most branc hes three form er students of Olkin who wrote dissertations on meta-analysis. They are Hedges ( 1980 ), Holmgren ( 1989 ) and Y en ( 1997 ). In addition, three other former stud en ts of Olkin are sho wn. Gleser, P erlman and S ampson eac h ha v e con tributed to the literature on meta-analysis or syn thesis of results, though n one w rote a disser- tation on the topic. Relev an t works include Gleser and Olkin ( 1994 , 1996 ), Koziol and Perlman ( 1978 ) and O lkin and Sampson ( 1998 ), among others. The next set of lea ve s shows student s of Olkin’s student s—p erhaps w e can call these O lkin’s meta- analytic “grandchildren.” Here are listed seve n who wrote dissertations on meta-analytic metho ds. Abu- Lib deh ( 1984 ), Bec k er ( 1985 ), Champ ney ( 1983 ), Konstan top oulos ( 2003 ), Pigott ( 1992 ) and Zhang ( 1993 ) w ere dissertations written b y studen ts of Hed- ges, and Sylv ester ( 2001 ) and Sezer ( 2006 ) we re d is- sertations directed by Gleser. Tw o other student s of Hedges (V ev ea and F riedman) ha v e contributed to the literature on meta-analytic metho ds after com- pleting a d issertation using meta-analytic metho d s or on another topic (e.g., F riedman, 1989 , 2000 ; Hedges and V evea , 1996 , 1998 ). Finally w e reac h the current end s of the branches. Six additional stud en ts are listed who work ed w ith Bec ker on meta-analytic metho ds (Chang, 1992 ; Chiu, 1999 ; Cho, 2000 ; F ahrbac h, 2001 ; Sc hram, 1996 ; W u, 2006 ) and t wo who were studen ts of V ev ea and who hav e either written diss ertations on meta- analysis metho ds (Hafdahl, 2001 ) or cont ribu ted to the meta-analytic literature (V ev ea and W o o ds, 2005 ; W o o ds et al., 2002 ) while wr iting a d issertation on a differen t topic. W e can b e assured that others will follo w. 3. O VERVIEW OF MUL TIV ARIA TE MET A-ANAL Y SIS W e next tur n to the topic of m ultiv ariate meta- analysis and explore O lkin’s fund amental contribu- tions to this domain. (See Beck er, 2000 , and v an Hou w elingen et al., 2002 , for ov erviews of the topic of m ultiv ariate meta-analysis.) Multiv ariate meta- analysis o ccur s when m ore than one (dep en d en t) outcome is measur ed in a study . This can o ccur when sub jects are measured on s ev eral outcomes or at sev er al time p oin ts (multiple end p oint studies), or when study in dices are computed using shared treat- men t or con trol groups (multiple treatmen t studies). These cases do not typical ly includ e studies with re- sults for multiple samples. While suc h samp les may exhibit sub tle dep endencies b ecause of common in- strumenta tion, treatmen ts and the lik e, their out- comes do not hav e a correlation structure that is eas- ily charact erized. Hedges and Olkin ( 1985 ) fi rst pr e- sen ted metho ds for dealing with m ultiv ariate data in m eta-analysis. Their Chapter 10 dealt with stan- dardized mean differences th at are dep end en t b e- cause p (dep end ent) resp onse v ariables are observed within eac h p rimary study . W e denote the results as T ij , where i indexes the study and j the outcome. Across studies w e may ha v e           T 11 . . . T 1 p T 21 . . . T 2 p . . . . . . T i 1 . . . T ip . . . . . . T k 1 . . . T k p           for k studies and u p to p outcome indices. The p de- p end ent indices arise wh en p resp onse v ariables are observ ed, w h en contrasts are dep endent (e.g., com- mon con trols, multiple prop ortions), w hen m ultiple indices inv olve eac h r esp onse v ariable (e.g., correla- tion matrices), and when multiv ariate analyses ap- p ear within a primary stud y . Th e p ossible metrics include m ultiv ariate standardized mean differences, correlations an d pr op ortions (or o d ds r atios). Each suc h metric will b e considered in turn . 4. MUL TIV ARIA TE ST ANDARDIZED MEAN DIFFERENCES This metric ma y b e the m ost thoroughly in ves- tigated of all those for wh ic h multiv ariate analy- ses ha v e b een prop osed. Gleser and O lkin ( 1994 ) dealt with m ultiple treatment studies and m ultiple endp oin t stud ies for standardized mean differences. Some s tu dies com bine b oth of these m ultiv ariate as- p ects. Evidence that m ultiv ariate effect-size data are common is found in the fact that Gleser and Olkin ( 1994 ) has b een cited o ver 100 times, in fields suc h as psyc hology , edu cation, m edicine, ecology and crim- inal justice. Similarly , an early pap er b y Rauden- bush, Bec k er and Kalaian ( 1988 ) dealt w ith multi- v ariate standardized-mean-difference data. 4.1 Multiple T reatment Studies Multiple treatmen t stu dies are illustrated h ere with an example of studies with a common con trol group. F urther elab orations of this scenario (e.g., 4 B. J. BECKER with three or more treatmen t group s or multiple con trol group s) lead to more outcomes, but the p rin- ciples underlying these m etho ds can b e illustr ated with th is simplest scenario. Supp ose a stu d y has t wo treatmen t grou p s, T 1 and T 2 , and one con trol group C . Then if we define ¯ X A to repr esent the mean of group A and S to b e the p o oled within-group s standard deviation across all groups, we can compute T 1 = ( ¯ X T 1 − ¯ X C ) /S a nd T 2 = ( ¯ X T 2 − ¯ X C ) /S for ea c h stu dy . If we in d ex these outcomes as T i 1 and T i 2 with i for the i th study , we will ha v e           T 11 T 12 T 21 T 22 . . . . . . T i 1 T i 2 . . . . . . T k 1 T k 2           whic h h as a m ultiv ariate structure. Gleser and Olkin ( 1994 ) ga v e t wo formulas for Cov( T ij , T ij ′ ) f or m ul- tiple treatmen t stud ies. More recen t w ork by Co ok ( 2004 ) p resen ts a formula tailored to s m all-sample cases. 4.2 Multiple Endp oint Studies Gleser and O lkin ( 1994 ) also cov er dep endence of standardized mean differences d ue to m ultiple r e- sp onse v ariables (expanding on Hedges and Olkin, 1985 ). If we define T ij to r epresen t an effect size for outcome measure j ( j = 1 to p ) in study i , we ha ve T ij = ( ¯ Y T ij − ¯ Y C ij ) /S ij for i = 1 to k studies and j = 1 to p measures. This was lab eled the multiple endp oint design. The effect-size data structure is id en tical to that s h o wn ab o v e but the co v ariances b et wee n th e m ultiple ef- fects from eac h study differ f rom those in the m ul- tiple tr eatment case. 5. MUL TIV ARIA TE PROPORTIONS Less h as b een published on the multiv ariate meta- analysis of prop ortions. One con tribution is Gleser and Olkin’s ( 2000 ) c hapter on m ultiple treatmen t studies with outcomes expr essed as tw o-by-t w o ta- bles. Gleser and O lkin presen t large-sample gener- alized least squares metho d s for deal ing with risk differences, log o dds ratios, and arcsine transformed prop ortions from multiple treatment stud ies. Other relev an t referen ces include Ar ends, V ok o and Stijn en ( 2003 ) and Nam, Mengersen and Garthw aite ( 2003 ) whic h concern analyses of multiple log-od ds ratios. Additional forth coming w ork will und oubtedly ad- dress this issu e. 6. MUL TIV ARIA TE CORRELA TIONS AND SLOPES The topic of synthesis of correlation matrices has seen increasing activit y in the p ast few yea rs. This increase in inte rest is lik ely related to the in creas- ingly complex m o dels inv estigated in pr imary re- searc h , at least in the so cial sciences. Researc hers w an t to b e able to statistica lly mo del the effects of m ultiple predictors as w ell as to con trol for p oten- tial confounding v ariables, and th is is done by in- cluding su c h v ariables in complex mo dels. Results of such tec h niques as structur al equation mo deling, factor analysis and m ultiple r egression ha v e often b een omitted from meta-analyses b ecause of a lac k of metho d s f or synthesiz ing ind ices from these an aly- ses. While Olkin has not contributed directly to this area of syn thesis metho ds, his work is fu ndamenta l b ecause most of the analyses prop osed to date are asymptotic and rely on the large-sample distrib u tion theory pr esen ted by O lkin and Siotani in 1967. The multiv ariate work in this realm of meta-analysis has in vol ve d the synt hesis of correlation matrices, and the use of those summaries in fu rther mo deling of linear mo dels, stru ctural equation mo dels, and ev en factor analysis (G. Bec ker, 1996 ). B. Bec ker and h er collab orators (B. Bec k er, 1992 , 1995 ; Bec k er and F ahrbach, 1994 ; Bec k er and S c hram, 1994 ) b e- gan this stream of wo rk by presenting metho d s for the synthesis of correlation matrices, sp ecifically es- timates of mean matrices u nder fixed- and r an d om- effects mo dels and tests of the homogeneit y of the series of matrices un der review. A t r ou gh ly the same time, applications of lik e metho ds app eared in the p ersonn el psyc hology literature (e.g., Schmidt, Hunt er and Outerbridge, 1986 ). Bec k er also pr esented meth- o ds for estimating linear mo dels based on the mean correlation matrices and testing comp onen ts of those comp osite m o dels. Others ha ve pur sued this w ork and inv estigated the use of mean matrices with struc- tural equation mo d eling softw are (e.g., Cheung and Chan, 2005 ; F ur lo w and Beretv as, 2005 ). All of these w orks rely on the fundamental r esult deriv ed b y Olkin and Siotani ( 1976 , page 238) of the cov ariance among correlations from a single sample. Sp ecifi- cally , the large-sample co v ariance, σ ist,iuv , b et w een MUL TIV ARIA TE MET A-A NAL YSIS 5 p opulation correlations ρ ist and ρ iuv within study i is σ r ist ,r iuv = [0 . 5 ρ ist ρ iuv ( ρ 2 isu + ρ 2 isv + ρ 2 itu + ρ 2 itv ) + ρ isu ρ itv + ρ isv ρ itu − ( ρ ist ρ isu ρ isv + ρ its ρ itu ρ itv + ρ ius ρ iut ρ iuv + ρ ivs ρ ivt ρ ivu )] /n i , where n i is the sample size in study i and s , t , u and v index the v ariables within stud y i that are corre- lated. That is, ρ ist is the correlation b et w een v ari- ables X s and X t within stu dy i . This result was also used by Hafdahl ( 2001 ) who examined exploratory factor analysis metho ds based on syn thesized ma- trices, and pap ers b y Olkin and other collab orators (e.g., O lkin and Finn, 1976 , 1990 ; Olkin and Saner, 2001 ) also rely on this fundamental r esult. 7. CONCLUSION It is safe to s ay that m uc h of the w ork on meta- analysis, and esp ecially m ultiv ariate iss u es in meta- analysis, has its genesis in the con tributions of In- gram Olkin. The review of researc h in this pap er sho ws the significan t impact of Olkin’s w ork. The family tree illustrates that contributions from Olkin’s academic descendan ts are numerous and will con- tin ue to b e forthcoming. A CKNO WLEDGMENTS This work w as supp orted b y National Science F oundation Gran ts REC-03356 56 and REC-06340 13. P arts of this w ork w ere p resen ted in the symp o- sium “Multiv ariate Analysis: In C elebration of In- gram Olkin’s 80th Birthd a y ,” at the Join t Statistical Meetings, T oron to, Canada, August 2004. REFERENCES Abu-Libdeh, O. I. (1984). Strength of association in the sim- ple general linear model: A comparativ e study of Hays’ omega-squared. Ph.D. dissertation, Univ. Chicago. Arends, L. R., Vok ˇ o, Z. and Stijnen, T. (2003). Com- bining multiple outcome measures in a meta-analysis: A n application. Statistics in Me dicine 22 1335–1353. Becker, B. J. (1985). App lying t ests of com bined signifi- cance: Hyp otheses and p ow er considerations. Ph.D. d isser- tation, Un iv. Chicago. Becker, B. J. (1992). Using results from replicated studies to estimate linear mo dels. J. Educ ational Statistics 17 341– 362. Becker, B. J. (1995). Correction to “Using results from replicated stud ies to estimate linear mod els.” J. Educ a- tional and Behavior al Statistics 20 100–102. Becker, B. J. (2000). Multiv ariate meta-analysis. In Hand- b o ok of Appli e d Multivariate Stat istics and Mathematic al Mo deling (H. E. A. Tinsley and S. D. Brown, ed s.) 499– 525. A cademic Press, San D iego, CA. MR1823752 Becker, B. J. and F ahrbach, K. (1994). A comparison of approac hes to the synthesis of correlation matrices. Paper presented at the annual meeting of the American Edu ca- tional Research Asso ciation, New Orleans, LA . Becker, B. J. and Schram, C. M. (1994). Examining ex- planatory mo dels through research synthesi s. In The Hand- b o ok of R ese ar ch Synthesis (H. M. Coop er and L. V. Hedges, eds.) 35 7–381. Ru ssell Sage F oundation, New Y ork. Becker, G. (1996). The meta-analysis of factor analyses: A n illustration based on the cumulation of correlation matri- ces. Psycholo gic al Metho ds 1 341–353. Champney, T. F. (1983). Ad justments for selec- tion/publication bias in quan titative research. Ph.D. dissertation, U niv. Chicago. Chang, L. (1992). A p ow er analysis of t h e test of homogene- it y in effect-size meta-analysis. Ph.D. dissertation, Mic hi- gan St ate Univ. Cheung, M. W. L. and Chan, W. (2005). Meta-analytic structural equation mo deling: A tw o-stage approach. Psy- cholo gic al Metho ds 10 40–64. Chiu, C. W .-T. (1999 ). Scoring p erformance assessmen ts based on judgements: Ut ilizing meta-analysis to estimate v ariance comp onents in generalizabilit y theory for unbal- anced situations. Ph.D. dissertation, Michigan State Un iv . Cho, J. M. (2000). Hyp otheses and limiting distributions for p-v alue summaries to compare studies in meta-analysis. Ph.D. d issertation, Michigan St ate Univ. Cook, S. R. (2004). A note on testing for homogeneity among effect sizes sharing a common control group. Psycholo gic al Metho ds 9 446–452. F ahrba ch, K. R. (2001). An inv estigation of metho ds for mixed-mo del meta-analysis in the presence of missing data. Ph.D. d issertation, Michigan St ate Univ. Friedman, L. ( 1989). Mathematics and the gender gap: A meta-analysis of recen t studies on sex differences in mathe- matical tasks. R eview of Educ ational R ese ar ch 59 185–213 . Friedman, L. (2000). Estimators of rand om effects vari ance compon ents in meta-analysis. J. Educ ational a nd Behav- ior al Statistics 25 1–12. Furlo w, C. F. and Beretv as, S . N . ( 2005). Meta-analytic metho d s of p o oling correlation matrices for structural equation modeling under different patterns of missing d ata. Psycholo gic al Metho ds 10 227–254. Glass, G. V . (1976). Primary , secondary , and meta-analysis of research. Educ ational R ese ar cher 5 3–8. Gleser, L. J. and Olkin, I. (1994). S to chastically dep endent effect sizes. In The Handb o ok of R ese ar ch Synthesis (H. M. Co op er and L. V. Hed ges, eds.) 339–355. Russell Sage F oundation, New Y ork. Gleser, L. J. and Olkin, I. (1996). Models for estimating the num b er of unpub lished stu dies. Statistics i n Me di cine 15 2493–2507. 6 B. J. BECKER Gleser, L. J. and Olkin, I. (2000). Meta-analysis for 2 × 2 tables with multiple treatment group s. In Meta-A nalysis in Me dicine and He al th Policy (D . A. Berry and D. K. Stangl, eds.) 339–355. Dekker, New Y ork. Hafdahl, A. R. ( 2001). Multiva riate meta-analysis for exploratory fa ctor analytic researc h. Ph.D. dissertatio n, Univ. N orth Carolina at Chap el H ill. Hedges, L. V. (1980). Com bining th e results of exp eriments using different scales of measurement. Ph.D. dissertation, Stanford Univ. Hedges, L. V. and Olkin, I. (1980). V ote-counting methods in research synthesis. Psycholo gic al Bul letin 88 359–369. Hedges, L. V . and Olkin, I. (1985). Statistic al Metho ds for Meta-A nalysis . A cademic Press, Orland o, FL. MR0798597 Hedges, L. V. and Vevea, J. L. (1996). Estimating effect size under publication bias: Small sample prop erties and robustness of a random effects selection mod el. J. Educ a- tional and Behavior al Statistics 21 299–332. Hedges, L. V. and Vev e a, J. L. ( 1998). Fixed- and random- effects models in meta-analysis. Psycholo gic al M etho ds 3 486–504 . Holmgren, E. B. (1989). U tilizing p–p plots in meta-analysis as general measures of treatment effects. Ph.D. disserta- tion, St anford Un iv. Ko nst antopoulos, S. (2003). V ariance comp onents estima- tion in meta-analysis emplo ying maximum likel iho o d meth- ods. Ph.D. dissertation, Univ. Chicago. Ko ziol, J. A. and Perlman, M. D. (1978). Combining in - dep endent chi-squared tests. J. A mer. Statist. Asso c. 73 753–763 . Nam, I. S., Me ngersen, K. and Gar thw aite, P. (2003). Multiv ariate meta-analysis. Statistics i n Me dicine 22 2309– 2333. Olkin, I. and Fi nn, J. D. ( 1976). Correlations redux. Psy- cholo gic al Bul letin 118 155–164 . Olkin, I. and Finn , J. D. (1990). T esting correlated correla- tions. Psycholo gi c al Bul l etin 108 330–333. Olkin, I. and Sampson, A. (1998). Comparison of meta- analysis versus analysis of vari ance of individual patien t data. Bi ometrics 54 317–322 . MR1626809 Olkin, I. and Sane r, H. (2001). Approximations for trimmed Fisher pro cedures in researc h synthesis. Statistic al Metho ds in Me dic al R ese ar ch 10 267–276. Olkin, I. and Siot ani, M. (1976). Asymptotic d istribution of functions of a correlation matrix. In Essays in Pr ob abil ity and Statistics Chapter 16 (S . Ikeda, ed.) 235–251. S hinko Tsusho, T okyo. MR0603847 Pearson, K. (1904). Rep ort on certain enteric fev er ino cula- tions. British Me di c al Journal 2 1243–1246 . Pigott, T. D. (1992). The application of normal maximum lik eliho od meth od s to missing d ata in meta-analysis. Ph.D. dissertation, U niv. Chicago. Raudenbush, S. W., Be cker, B. J. and Kalaian, H. (1988). Modeling multiv ariate effect sizes. Psycholo gic al Bul letin 103 111–12 0. Schmidt, F., Hunter, J. E. and Outerbridge, A. N. (1986). Impact of job ex p erience and abilit y on job knowl- edge, wo rk sample p erformance and sup erv isory ratings of job p erformance. J. A pplie d Psycholo gy 71 432–439. Schram, C. M. (1996). A comparison of method s for cor- recting multiv ariate data for attenuation with application to syn thesizing correla tion matrices. Ph.D. d issertation, Mic higan State Univ. Sezer, A. (2006). R ep orting uncertaint y by spline function approximatio n of log-likelihood . Ph.D. d issertation, Un iv . Pittsburgh. Syl ve ste r, M. A., I I. (2001). Estimation of a common mean through a series of simi lar interlaboratory exp eriments. Ph.D. d issertation, Un iv. Pittsburgh. v an Houwelingen, H. C ., Arends, L. R. and S tijnen, T. (2002). T utorial in biostatistics. A dv anced metho ds in meta-analysis: Multiv ariate approach and meta-regression. Statistics in Me dicine 21 589–624. Vevea, J. L. and Woods, C . M. (2005). Publication bias in researc h synthesis: Sensitivity analysis using a p riori weigh t functions. Psycholo gic al Metho ds 10 428–443. Woo ds, C. M., Vevea, J. L., Chambless, D. L. and Ba yen, U. J. (2002). Are compulsive chec kers impaired in memory? A meta-analytic review. Cl inic al Psycholo gy : Scienc e and Pr actic e 9 353–366. Wu, M. J. (2006 ). Metho ds of meta-analyzing regression studies: Applications of generalized least squares and fac- tored lik elihoo ds. Ph.D. dissertation, Michiga n S tate Univ. Yen, J. H. (1997). Robust estimation of effect sizes in meta- analysis. Ph.D. dissertation, Stanford Univ. Zhang, J. (1993). Statistical significance pu blication bias, its determination and statistical adjustments in meta-analysis. Ph.D. d issertation, Un iv. Chicago.