Comment: Struggles with Survey Weighting and Regression Modeling

Statistic al Scienc e 2007, V ol. 22, No. 2, 168– 170 DOI: 10.1214 /0883423 07000000195 Main article DO I: 10.1214/0883 42306000000691 c  Institute of Mathematical Statisti cs , 2007 Comment: Struggles with Survey W eighting and Regression Mo deling F. Ja y Breidt and Jean D. Opsomer W e congratulate the author o n a n informativ e and though t-pro vo king discussion o n a topic of broad in terest to the sta tistics comm un ity: the fitting of mo dels to data collected through complex surveys. The num b er of pap er s wr itten on this topic, whether from a mo d el-based or design-based p ersp ec tiv e, is substanti al and goes bac k at least to K on ij n ( 1962 ). This topic has led t o some disagreemen ts b et we en those adv o cating that the desig n b est b e ignored when the primary interest is on the c haracteristics of the mo del, and those s tating that the design cannot b e ignored. More recen tly , b oth sides of this d iscus- sion ha v e mo ved to something app roac h ing a con- sensus, with those fa vo ring a mo d el-based approac h ac kn o w ledging the need to account for nonignorable designs in the mo del fitting, w h ile the traditional design-based view has b een extended to explore cer- tain circumstances under which it is app ropriate to ignore the design. The cu rrent article is an excellen t example of those recen t discussions of why the design n eeds to b e ac- coun ted for in mo deling, and ho w this can b e done in practice. Th e imp ortance of ful ly accounti ng for the design b y in corp orating all relev an t in teractions pro vides a go o d motiv ation for the discussion of the range of metho ds in the article. It al so stresses other asp ects of imp ortance to p eople working with survey data, in particular the d esirabilit y of main taining scale/location in v ariance and linearit y of the mo del- based estimators. This ensur es consistency of esti- mates for different v ariables in the surv ey , as w ell as F. Jay Br eidt is Pr ofessor and Chair, Dep artment of Statistics, Color ado S tate University, F ort Col lins, Color ado 80523, USA e-mail: jbr eidt@stat.c olostate.e du . Je an D. Opsomer is Pr ofessor, Dep artment of Statistics, Iowa State University, Ames, Iowa 50011, USA e-mail: jopsomer@iastate.e du . This is a n elec tr onic reprint of the original a rticle published b y the Institute of Mathematica l Statistics in Statistic al Scienc e , 20 07, V ol. 22, No. 2, 168– 170 . This reprint differs from the original in pag ination and t yp ogr aphic detail. additivit y ov er d omains within th e p opulation. (As an aside, the p oststratified estimator arising from logistic r egression in S ection 3.2 can b e mo d ified to yield approxima te weigh ts b y the metho d p rop osed in W u and Sitter, 2001 .) The article ment ions a num b er o f disadv an tages of design-based (w eigh ted) mo del fitting and infer- ence. W eigh ts are viewed as complicated and m ys - terious, in the sense that th e mo deler often do es not kno w h o w they were constructed and hence migh t not w an t to rely on them when it comes to mo del sp ecification and estimation. Estimation, and esp e- cially v ariance estimation, are view ed as more cum- b ersome under the design-based paradigm compared to a mo del-based analysis. In what f ollo ws, w e will argue that a w eighte d analysis offers some distinct adv an tages and might actually redu ce the complex- it y of the analysis in many cases, at least from the p ersp ectiv e of a statistician inte rested in using pre- viously collected and w eighte d su rv ey data to fit a mo del. A key feature of the design-based paradigm (broad- ly sp eaking) is that it makes it p ossible to separate design and p ostsample adjustmen ts from data anal- ysis. Individu als task ed w ith creating survey w eigh ts are typica lly w ithin the organization col lecting the data, and will b e referred here as “the survey statis- ticians.” They ha v e kno wledge of the sampling de- sign and h a v e access to detailed information on the nonresp onse c haracteristics of the sample and to rel- ev an t auxiliary information. Based on these sources of information, they d evelo p a set of su rv ey weig hts (and sometimes also pro du ce sets of replication w eigh ts for v ariance estimation). As noted in the article, these weigh ts are often m uc h more compli- cated than simple in v erses of inclusion probab ilities, and in fact refl ect the b est effort on the p art of the surve y statisticians creating the weigh ts to acco unt for non r esp onse and incorp orate p oten tially usefu l p opulation-lev el information. T hese we igh ts are ap- p end ed to the dataset, whic h is then made a v ailable to individu als int erested in analyzing those d ata. 1 2 F. J. BREIDT AND J. D. OPS OMER These individuals will b e referred to as “the data analysts.” F rom the p ersp ectiv e of the data analysts, using these w eight s is con venien t in the sense that they pro vide a simple wa y to accoun t for the w a y the data were obtained, without requ iring the d ata an- alysts to replicate many of th e tasks of the survey statisticia ns. O v erall, this “division of lab or” allo ws b oth sets of statisticians to fo cus their efforts on the p ortion of the ov erall problem of most immed iate in terest to them, and f or wh ic h th ey hav e b oth the exp ertise and the in f ormation a v ailable to b est p er- form the requ ired tasks. As noted by a num b er of authors (e.g., Pfeffer- mann, 1993 ), p erformin g a w eight ed analysis for a mo del using inv erses of the inclusion pr obabilities ensures that th e resu lting estimators are d esign con- sisten t for p opulation-lev el q u an tities, wh ic h are themselv es mo d el consisten t for the mo del parame- ters of interest. When the w eigh ts also in clude non- resp onse adjustments (usually by wa y of p oststrat- ification) a s w ell as other calibration ad j ustment s, results for descriptiv e statistics, in cluding those dis- cussed in S¨ arndal and Lundstr¨ om ( 2005 ), sho w that the estimators are consisten t u nder the joint design- resp onse m ec h anism. While these results are exp ect- ed to con tinue to hold when mo del parameters are targeted rather than fi nite p opulation means, there is currently only limited formal theory exp loring this topic. The d ivision o f lab or bet wee n the surv ey s tatis- ticians and the data analysts has some additional adv an tages. Wh ile the former t ypically ha ve access to detailed unit-lev el information and can use that information in the construction of th e w eigh ts, con- fidenti alit y issu es often preclude such access for th e latter. F or instance, in the S o cial I n dicators S ur- v ey considered in the Gelma n article, av oiding the w eigh ts required kn o wledge of the n umber of adults and the n u m b er of phone lines in the household of eac h resp ondent, as w ell as v arious other demo- graphic v ariables. It is easy to envision situations where at least some of these v ariables are not made a v ailable to the data analysts in order to protect the confident ialit y of the s urve y resp ondents. In such sit- uations, the data analysts could still try to b uild a mo del that incorp orates the d esign effects, but migh t end up only b eing partly successful b ecause some influentia l v ariables are not a v ailable. Another consider ation is the fact that large-scale surve ys often inv olv e complex stratification and p ost- stratification sc hemes, multiple phases and/or stages of selection, imputation for item nonresp onse, etc. Accoun ting for all these factors, ev en if the needed sources of information are a v ailable to th e data an- alysts, w ould require significant time and effort on the p art of the d ata analysts and result in m o d- els that m igh t b e unwieldy and difficult to in ter- pret. One p oin t noted in the Gelman article is that v ariance estimation for weig hted estimators is more cum b er s ome than for fu lly mo del-based estimators. T o a large exten t, this is indeed the case, bu t a num- b er of s olutions are a v ailable. F or sp ecific mo d els (e.g., linear or logistic regression), commercial soft- w are programs s uc h as SAS are in cr easingly provid- ing design-based estimation pro cedu res, so that with access to th e weigh ts and some b asic inf ormation ab out the design (e.g. , strati fication in formation and primary samp ling unit identifiers), it is p ossible for the data analysts to p erf orm design-based inference for mod el parameter estima tors. An al ternativ e pro- cedure, already alluded to earlier and often used for large-scal e surveys, is for the su rv ey statisticians to pro vide sets of replicatio n weig hts (e.g., j ackknife or b o otstrap replicates). In that case, v ariance estima- tion f or t he w eigh ted estimates is a simple ma tter of recomputing the estimates for eac h s et of repli- cate w eigh ts and calculating the v ariabilit y among the rep licate estimates. Incorp orating the design and nonresp onse c har- acteristics of a dataset through explicit mo deling is a s tatistica lly v alid and conceptually attractiv e ap- proac h to solving the nonignorabilit y problem. It h as the adv an tage of b eing easily in tegrated into the set of to ols most familiar to data analysts, but, as ex- plained in th is interesting article, it requires knowl- edge of the relev ant v ariables and has to b e d one carefully . Performing a design-based analysis with the w eigh ts pro vided as part of a su rv ey dataset is attractiv e as well , b ecause it is generally applicable ev en withou t deta iled kno wledge of the wa y the data w ere obtained. In closing, w e w ould lik e to s uggest a num b er of p ossible dev elopmen ts that w ould help mak e d ata analysts more comfortable w ith these we igh ted anal- yses. While weig ht constru ction is lik ely to remain to a large exten t an “art,” more transparency in how w eigh ts are constructed migh t alleviate some of the discomfort on the part of data analysts h a ving to rely on the w ork of su rv ey statisticians as a bu ilding blo c k in their o wn analysis. A relate d dev elopmen t COMMENT 3 migh t b e more edu cation and tr ainin g in the int er- pretation of results of w eighte d analyses for nonsur- v ey statisticians and in metho ds for doing inference for design-w eigh ted mo del estimates. On the s u rve y statistics side, we wo uld like to encourage the in- v estigatio n of the statisti cal prop erties of w eighte d estimators for mo del p arameters th at explicitly ac- coun ts f or the m ultiple adjustmen ts t yp ically made to surve y we igh ts, including calibration and nonre- sp onse weigh ting. REFERENCES Ko nijn, H. S. (1962). Regression analysis in sample surveys. J. Amer. Statist. Asso c. 57 590–606. [Correcti on 58 (1963) 1162.] MR0148179 Pfeffermann, D. (1993). The role of sampling weig hts when mod eling survey data. Internat . Statist. R ev. 61 317–337. S ¨ arndal, C.-E. and Lun dstr ¨ om, S. (2005). Estimation in Surveys with Nonr esp onse . Wiley , Chic hester. MR2163886 Wu, C. and Sitter, R . R. (2001). A mod el-calibration ap- proac h to using complete auxiliary information from survey data. J. A mer. Statist. Asso c. 96 185–193. MR1952731