Discussion of: Statistical analysis of an archeological find

The Annals of Applie d Statistics 2008, V ol. 2, No. 1, 77–83 DOI: 10.1214 /08-A OAS99C Main articl e DO I: 10.1214/ 08-AOAS99 c  Institute of Mathematical Statistics , 2 008 DISCUSS ION OF: S T A TISTICAL A NA L YSIS OF AN AR C HEOLOGICAL FIND By Holger H ¨ ofling 1 and Larr y W asserma n 2 Stanfor d U niversity and Carne gie Mel lon University Ther e ar e no smal l c oincidenc es and big c oincidenc es! Ther e ar e only c oincidenc es! F rom “The Statue” episo d e of Seinfeld . 1. Introdu ction. Andrey F euerverge r has un dertak en a s er ious challenge . The su b ject matter is con tro v ersial and findin g a sensib le wa y to formulat e the p roblem in a rigorous statistica l manner is difficult. The pap e r is notable for its th orou gh n ess. W e ha ve rarely seen a pap er on an applied problem that p ro vid es so m uc h bac kground material. Most imp ortantl y , the author is very careful to do cu men t all h is assumptions and to r emind the reader that the conclusion is sensitiv e to these assumptions. He resists the temptati on to p resen t his r esults in a sensationali stic w a y . Rather, he con v eys his analysis in a dispassionate, understated tone. Nonethele ss, he could still end up on Opr ah . W e are tryin g to assess the pr obabilit y of a hyp othesis when the hyp othesis is formed after seeing the d ata. This is a notoriously difficult pr oblem. As F euerv erger notes, coincidences are common. But just h o w common? One resp on s e—the nihilistic appr oac h—is to say that it is imp o s s ible and stop there. W e ha v e m uc h sympathy with the nihilists in a problem lik e this. Perhaps the scienti fically honorable path is to say that any answ er is m isleading so it is b etter to p ro vid e no answe r. But ultimately th is is unsatisfying and we accept the author’s approac h to pr o vide an analysis with many ca veat s. The question ma y b e f r amed formally as f ollo ws. W e observe an outcome x —a tom b with in teresting n ames—and we w ant to kno w : is this outcome Received Sep tember 2007 . 1 Supp orted b y an Albion W alter Hewlett Stanford Graduate F ello wship. 2 Supp orted by NS F Grant CCF-06-25879 . Thanks to R ob Tibshirani and I sa V erdinelli for helpful commen ts. This is an elec tronic r eprint of the o riginal a r ticle published by the Institute of Mathematical Statistics in The Annals of Applie d Statistics , 2008, V ol. 2, No. 1 , 77–8 3 . This reprint differs fro m the original in pagina tio n and t ypo graphic detail. 1 2 H. H ¨ OFLING AND L. W AS S ERMAN surpr ising? One wa y to quanti fy sur p risingness is to p erform the follo wing steps: 1. Cons truct a sample s p ace X th at con tains x . 2. Id en tify all the outcomes A that wo uld h a ve b een considered surpr ising if they had b een obs er ved. 3. Cons truct an appropriate null distribu tion P 0 . 4. Comp ute the p -v alue p = P 0 ( A ). The most difficult step is iden tifying the set A of inte resting outcomes. It is explicitly counte rfactual to ask if an outcome would hav e b een s urpr isin g if it had o ccur r ed, knowing that it did not o ccur. 2. F euerverg er’s app roac h. What the author has p rop osed is b oth in ter- esting and reasonable. Numerous judgemen t calls h a ve to b e m ade b u t they ha ve b een carefully do cumented. Our sum mary of F euerv erger’s m etho d is this: The sample space is c h osen to b e sets of names on ossu aries, sub ject to some restrictions. Th e n u ll measure is essentia lly r an d om sampling from an onomasticon. The auth or defines a statistic (RR) that maps sets of names in to pro ducts of n u mb ers. These n u m b ers are essent ially sample p r op ortions, mo dified to tak e into acco unt v arious nuances s u c h as sur prisingness of ver- sions of names. The resu lt is a v ery small p -v alue s u ggesting that the find is indeed su rprisin g. The ‘Mariamenou η M ara’ inscription has a very big effect on F euerv erger’s RR statistics. An explanation for this is that the RR statistic b ecomes more significan t if broad name categories are b eing sub divided in to sp ecial name renditions, ev en if the p articular name rend itions are not relev ant. The fol- lo wing example illustrates this p oin t: A p opulation has th ree names A , B and C eac h with frequency 1 / 3. A has 2 name renditions A 1 (1 / 3 of A ) and A 2 (2 / 3 of A ) . Ou r family has tw o members named A and B , and A 1 and A 2 are b oth relev ant. The uncov ered tom b has one inscription A 1 . When only considering broad name categories , w e have RR ( A ) = 1 / 3, RR ( B ) = 1 / 3 and R R ( C ) = 0. When the null is random draw in g from the population, the p -v alue is then 2 / 3. When taking name renditions into account, R R ( A 1 ) = 1 / 9, RR ( A 2 ) = 2 / 9 , RR ( B ) = 1 / 3 and R R ( C ) = 0 giving p -v alue of 1 / 9. The p - value decreased although b oth name renditions w ere considered relev ant. The c hange in p -v alue can b e ev en more substanti al in more complic ated cases. In this commen t, w e present a F requen tist and a Ba ye s ian approac h that do not h a ve this problem and y ield quite differen t results. 3. A d ifferen t approac h. W e wo uld lik e to consider a d ifferent wa y of defining the basic eve n t A . Our appr oac h is more expansive and, as a result, more conserv ativ e. Instead of asking “What is the pr obabilit y of getting this DISCUSSI ON 3 set of names?” w e ask “What is the pr obabilit y of getting some in teresting set of names if one lo oks at s ev eral tom b s?” Let X b e all name sets. Examples of sample p oin ts in X are x = { Salome } , x = { Levi , Hanan , Simon , Mariam } , x = { Joseph , Jesus , Sarah } , and so on. Define a list of target names S . The list should includ e all names that will spark in terest. W e tak e this to b e either the big set S = { Mariam , Mary , Salome , James , Joseph , Joanna , Martha } or the small set S = { Mariam , Mary , Salome , James , Joseph } . The name “Jesus” is not includ ed b ecause w e will treat it separately . W e assume that a tomb would ha ve triggered int erest if its name s et B has sufficien t o v erlap with S . W e lum p together different version of names since in terested observ ers wo u ld surely argue that a tom b is in teresting if there is an y w a y at all of m atching the found names to p otentia lly int eresting names. Denote the name sets in the tom bs by B 1 , . . . , B N . Sa y that B i is in teresting if | B i ∩ S | ≥ 3 and “Jesus” ∈ B i . W e denote th e probability of this even t by π i . Assumin g indep end ence of name assignmen ts in and across tom bs, the p -v alue is p = 1 − N Y i =1 (1 − q ( n i , π i )) where n i is the n u m b er of ossuaries in tom b B i , q ( n i , π ) = p J P ( Y i ≥ 3) , Y i ∼ Binomial( n i − 1 , ν ) , ν is th e probability that a single n ame dra wn at r andom is in S and π J is the probability of dra wing the name “Jesus.” W e d o not tak e π J to b e the probabilit y of dra wing “Jesus son of Joseph ” b ecause th e tom b could ha v e b een considered inte r esting if it had only said “Jesus.” F or our cal culations w e tak e N = 100, n i = 6. The n umber 100 co m es from the fact th at there are ab out 1000 tombs bu t only 10 p ercen t ha ve b een exca v ated. Hence π i = π d o es not v ary w ith i . W e consider t wo p ossibilities for the m ale-femail r atio: (i) equal or (ii) unequal as represente d by the 4 H. H ¨ OFLING AND L. W AS S ERMAN onomasticon. F or example, in case S is equal to the fir s t (big) c hoice, the male/female ratio is equal we get ν = 1 2  231 + 103 + 45 2509  + 1 2  81 + 63 + 21 + 12 317  = 0 . 354 7 . The v alue of π and p for the different combinatio ns of assump tions is as follo ws: S m / f ratio π p -v alue big equal 0.005 0.393 big not equal 0.002 0.183 small equal 0.00 3 0.290 small not equal 0.00 2 0.158 W e r eiterate that w e hav e n ot treated n ame v ariations as sp ecial. But the calculation is in v ariant under splitting names into sub catego ries since w e are fin ding the p r obabilit y of a set of in teresting n ames, not a p articular name. W e also ignored family structure. W e no w consider tw o v ariations. W e consider replacing “Jesu s” with “Jesus son of J oseph” by m u ltiplying these t w o probabilities. W e also c onsider taking N = 1000 to reflect the unobserved tom bs. The r esults are: N = 100 N = 1000 Jesus 0.16 0.82 Jesus s on of Joseph 0.01 0.13 There is one case where the p -v alue is small. But the lac k of robu stness of th is result do es not mak e us confi den t in rep orting a small p -v alue. W e conclud e that the obs erv ed ev ent is not rare at all. T he chance that an observer would fin d a tom b that could b e said to con tain interesting target names is large. Th is is due to th e fact th at the interesting n ames are common and that the man y tom bs pr o vide many opp ortunities for apparen t surpr ises. 4. Bay esian analysis. No w w e consider a Ba ye sian analysis of the prob- lem. W e need to compute P ( θ = 1 | x ) = P ( x | θ = 1) P ( θ = 1) P ( x | θ = 1) P ( θ = 1) + P ( x | θ = 0) P ( θ = 0) , where x denotes the data, θ = 1 that the tom b is fr om the NT family and θ = 0 that the tom b is from the normal p opulation. In the frequen tist approac h, a partial ordering h as to b e defin ed on the space of all outcomes. F euerv erger do es this using th e RR statist ic and the approac h d escrib ed ab o ve uses intersectio n of name sets. How ev er, d iscerning the exact ordering on the space of outcomes ma y b e hard or p eople m igh t n ot DISCUSSI ON 5 agree with it. The adv an tage o f the Bay esian a pproac h is that the alternativ e distribution only h as to b e defined at the p oin t x and no ord ering on the space of p ossib le outcomes is needed. 4.1. P osterior pr ob ability. Let us in tro duce a little more notatio n at this p oin t. L et c b e the configuration of a tomb, g b e its genealogy , n = ( n 1 , . . . , n K ) the br oad name categories and r = ( r 1 , . . . , r K ) the particular name r enditions. Assuming that ev ery name rendition only dep ends on θ and its broad name catego ry , we can w rite P ( x | θ ) = P ( c, g | θ ) P ( n | g , c, θ ) K Y i =1 P ( r i | n i , θ ) . Simplifying assumptions: T o mak e the computations easier, we make tw o more assu mptions: 1. Th e configuration and genealogy we exp ect to see in the NT family tom b is n ot different fr om the rest of the p opulation, that is, P ( c, g | θ = N T ) = P ( c, g | θ = P ). 2. Th e particular name rend itions we exp ect to see in the NT family tom b are no d ifferen t than wh at we exp ect to s ee in the rest of the p opula- tion, that is, P ( r i | n i , θ = N T ) = P ( r i | n i , θ = P ) . This assumption will b e relaxed later. Then the p osterior o dds are P ( θ = 1 | x ) P ( θ = 0 | x ) = P ( θ = 1) P ( θ = 0) · P ( n | c, g , θ = 1) P ( n | c, g , θ = 0) . 4.2. D istributions. First, we d efine the prior distribu tion. F euerverger estimates the num b er of tom b s in the area to b e ab out N = 1100. Also, let the p rior p robabilit y of the NT family ha vin g a tom b at all b e t . Th en P ( θ = 1) = t 1 N , P ( θ = 0) = 1 − P ( θ = N T ) . In ord er to b e optimistic, we tak e t = 1 and get prior o dds of P ( θ = 1) P ( θ = 0) = 1 1099 . This p rior can b e thought of as a Bay esian approac h to accoun t for data sno oping, th at is, the p oten tial to searching through m any tom bs. F or the n u ll distrib ution, names are drawn randomly us in g the name frequencies in Ilan. Men and w omen are b eing treated s eparately and the list of names n is treated as unordered . When sp ecifying the probabilit y distribution und er the alternativ e, it is necessary to weig h flexibility aga inst complexit y . Here w e wan t to tak e the 6 H. H ¨ OFLING AND L. W AS S ERMAN T able 1 Weights for e ach of the p ersons liste d Jesus Scenario son of Joseph James Joses Matthew Judas Others Neutral 20 3 3 62 / 2509 171 / 2509 3 Optimistic ∞ 1 1 62 / 2509 171 / 2509 0 Mary a Mariam Salome Scenario (mother) (sister) (sister) Mary Magdalen e Others Neutral 10 3 3 3 3 Optimistic ∞ 1 1 0 0 Draw in g from the set is b eing done with probabiliti es prop ortional to t h e w eights without replacemen t. The w eigh t in the “others” category is the w eight for all not listed p ersons. follo wing approac h : Sp ecify a s et of n ames from the NT family (separately f or men and w omen) and assign eac h name a w eigh t as to ho w lik ely it is to find this p erson in the NT family tom b. Then , the probability of a sp ecific tom b is calculated b y drawing fr om the nameset without r eplacemen t according to the w eights. T he weig h ts can b e determined in an optimistic or more conserv ativ e fashion (see T able 1 ). F or simplicit y , the p r obabilit y o f b eing in the generational o ssuary is tak en to b e the same for ev eryone, un d er the null as w ell as the alternativ e. 1 Neutral scenario: In this case, we chose the w eights in a fashion that seemed r easonable to u s when we do n ot consider the information gathered from the tom b. Also, eac h n ame in th e tom b is tak en as its b road name catego ry and it is assum ed that no additional information for sp ecial name renditions is a v ailable for the NT family . Neutral with sp ecial rend itions: Here, w e use the s ame w eight s as in the neutral scenario, how ev er accoun t for the sp ecial “Mariamenou η Mara” rendition. Eac h of the other inscriptions on the ossuaries is n ot sp ecial, s o w e do not mak e any adjustments for those. A pr iori, we could n ot h a ve known the in scription “Mariamenou η Mara,” so ho w do w e accoun t for it? Under θ = P , w e assume that for the Marya name catego r y , th e probabilit y of seeing a new pr eviously unseen name is 1 / 80. F or θ = N T , w e assu me that sp ecial name rend itions are more like ly , say 1 / 10. Assuming that ‘Mariamenou η Mara’ could in some w a y b e interpreted for Maria (mother), Mariam (sister) and Maria Magdalena, this raises the o d ds b y a factor 8 o ver the neutral scenario. Optimistic scenario: W e also w anted to explore the effect of ha ving v ery optimistic assu mptions which are to a large degree influenced by what has 1 This may b e viewed as an o versimplification, ho wev er as the weigh ts provide ample opp ortunity to fi ne tune prior beliefs, w e do not see this a s practically important. DISCUSSI ON 7 T able 2 Posterior pr ob ability that the T alpiyot tomb b elongs to the NT fami ly under various sc enarios Scenario Probability Neutral 3.4% Neutral—sp ecial renditions 21.8% V ery o ptimistic 64.1% b een observ ed in the tomb. J esus and his mother are taken to be in the tom b for sure. F or the rest of the men, the we ights are equal for b oth brothers and set to the norm al name f r equency in the p opulation for Matthew and Judas. The o verall effect of this choi ce of weig hts is to effectiv ely ignore the Matthew and Jud as ossu aries, assum e that one of the ossu aries is fr om a brother and one from a sister of J esus and assign all eligible brothers and sisters the same weigh t. 4.3. R esults. Ev en in the optimistic scenario, ther e is only ab out a 60% c hance of the tom b b elonging to the NT family . In the other tw o, more realistic schemes, the probability is only 22% and 3% (see T able 2 ). Ju s t as F euerv erger, we also did n ot consider the generational part of the “Ju das, son of Jesus” ossu ary . Including it in the analysis would b e p ossible; ho wev er, as prior b eliefs ab out a p ossible son of Jesus are very str on g, this ma y ha v e o ve rwhelmed the r est of the analysis and therefore we decided to exclude it. 5. Conclus ion. When asked to analyze these data, w e susp ect that man y statisticia ns w ould ha v e said that the problem is to o v ague an d w ould ha ve stopp ed there. W e commend Andr ey F eurve rger for plunging in and d oing a serious analysis. Ou r analysis su ggests that the find ing do es not lend s up- p ort to the h yp othesis that the fin d is indeed the tom b of the NT family . Ultimately , schola rs of history and arc heology will judge the v alidit y of the claims ab out this find. Dep ar tment of St a tistics St anford Un iversity St anford, California 94305 USA E-mail: hhoeflin@stanford.edu Dep ar tment of St a tistics Carnegie Mellon Un iversity Pittsburgh, Pennsyl v a nia 15213 USA E-mail: larry@stat.cmu.edu