A method for comparing non-nested models with application to astrophysical searches for new physics

MNRAS 000 , 1 – 5 (2016) Preprint 22 F ebruary 2016 Compiled using MNRAS L A T E X style file v3.0 A metho d for comparing non-nested mo dels with application to astroph ysical searc hes for new ph ysics Sara Algeri 1 , 2 ? Jan Conrad 2 , 3 , 4 , 1 and Da vid A. v an Dyk 1 1 Statistics Section, Dep artment of Mathematics, Imp erial Col le ge L ondon, South Kensington Campus, London SW7 2AZ, Unite d Kingdom 2 Dep artment of Physics, Sto ckholm University, Alb aNova, SE-106 91 Sto ckholm, Swe den 3 The Oskar Klein Centr e for Cosmop article Physics, Alb aNova, SE-106 91 Sto ckholm, Swe den 4 Wal lenb er g A cad emy F el low Accepted 2016 F ebruary 10. Received 2016 F ebruary 10; in original form 2015 Decem b er 25. ABSTRA CT Searc hes for unkno wn ph ysics and decisions b et ween competing astroph ysical mo d- els to explain data both rely on statistical hypothesis testing. The usual approach in searc hes for new physical phenomena is based on the statistical Likelihoo d Ratio T est (LR T) and its asymptotic prop erties. In the common situation, when neither of the t wo models under comparison is a special case of the other i.e., when the h yp othes es are non-nested, this test is not applicable. In astrophysics, this problem o ccurs when t wo mo de ls that reside in different parameter spaces are to be compared. An important example is the recently rep orted excess emission in astrophysical γ -rays and the ques- tion whether its origin is known astrophysics or dark matter. W e dev elop and study a new, simple, generally applicable, frequent ist metho d and v alidate its statistical prop- erties using a suite of simulations studies. W e exemplify it on realistic simulated data of the F ermi-LA T γ -ra y satellite, where non-nested hypotheses testing appears in the searc h for particle dark matter. Key w ords: statistical – data analysis – astroparticle physics – dark matter. This is a pre-cop yedited, author-produced PDF of an article accepted for publication in MNRAS Let- ters following peer review. The v ersion of record A metho d for c omp aring non-neste d mo dels with ap- plic ation to astr ophysic al se arc hes for new physics , doi: 10.1093/mnrasl/slw025, is a v ailable online at: http://mnrasl.oxfordjournals .org/cgi/content/ abstract/slw025?ijkey=UY4DKz 87GlCpToU&keytype=ref . 1 MODEL COMP ARISON IN ASTR OP AR TICLE PHYSICS In astroph ysics, hypothesis testing is ubiquitous, b ecause progress is made by comparing comp eting mo dels to ex- perimental data. In the sp ecial case, where new physical phenomena are searched for, the most common choice of h yp othesis test is the Likelihoo d Ratio T est (LR T), whose popularity is partly motiv ated b y the fact that, assuming H 0 is true, the asymptotic distribution of the LR T statistic is a χ 2 . Such result holds if the regularity conditions sp ecifi ed in Wilks’s theorem hold ( Wilks 1938 ). A k ey necessary condi- tion is “nested-ness” , meaning that there is a full mo del of ? E-mail: s.algeri14@imperial.ac.uk whic h both the mo dels under H 0 and the alternative h yp oth- esis, H 1 , are sp ecial cases. This is obviously the case for the searc h for new particles where the null hypothesis (or base- line model), H 0 , is given by “bac kground” and H 1 is giv en b y “bac kground+signal of new particle” . How ever, cases where model comparison is non-nested are common: for instance, when a kno wn astro physical signal can be confused with new ph ysics, see Ac kermann et al. ( 2012 ) for an example from as- troparticle ph ysics, or if the mo dels to be compared reside in different parameter spaces ( Profumo & Linden 2012 ); as in gamma-ray bursts ( Guiriec et al. 2015 ). In these situa- tions, Mont e Carlo simulations of the measuremen t pro cess are often the only possibility , but are c hallenged b y string ent significance requirements, e.g., at the 5 σ lev el. W e present a solution that allows ev aluation of accurate statistical sig- nificances for non-nested mo del comparison while av oiding extensiv e Mon te Carlo simulations. As a concrete example, w e apply the proposed pro cedure to the searc h for particle dark matter, where the metho d has particular imp ortance. One wa y to searc h for dark matter is to consider its h yp othesized annihilation pro ducts, i.e., γ -ra ys, that can b e detected b y space b orne or ground based γ -ra ys telescop es ( Conrad, Cohen-T anugi & Stigari 2015 ). Here, the issue of source confusion is one of the most challenging asp ects of claiming discov ery of a dark matter induced signal. A de- c  2016 The Authors 2 S. Algeri et al. tected excess of γ -ra ys may either originate from dark mat- ter annihilation or b e caused by conv entional, known as- troph ysical sources. Discrimination can b e p erformed using their sp ectral distributions, how ever these are not necessar- ily part of the same parameter space (see below). This situa- tion arises for example in the search for dark matter sources among the uniden tified sources found b y F ermi-LA T ( Ack- ermann et al. 2012 ), the claimed detection of a signal consis- ten t with dark matter in our own galaxy , whic h has gained m uch attention recen tly ( Daylan et al. 2014 ), or (once a detection has b een made) in the searc h for dark matter in dw arf galaxies ( Ac kermann et al. 2011 , 2014 , 2015 ; Geringer- Sameth & Koushiappas 2011 ; Geringer-Sameth et al. 2015 ). In the recen t claims, the existence of a source of γ -ra ys (ov er some bac kground) is established b y a LR T, but the crucial and unsolved question is not whether a γ -ra y source exists, but whether it can be explained by conv entional sources of γ -ra ys as opp osed to dark matter annihilation. This is a prime example of an non-nested mo del comparison. F or def- initeness, we can assume f ( y , E 0 , φ ) ∝ φE φ 0 y − ( φ +1) is the probabilit y density function (p df ) of the γ -rays energies, denoted by y , originating from known cosmic sources and g ( y, M χ ) ∝ 0 . 73  y M χ  − 1 . 5 exp  − 7 . 8 y M χ  is the p df of the γ -ra y energies of dark matter ( Bergstr ¨ om, Ullio & Buckley 1998 ). The goal is to decide if f ( y , E 0 , φ ) is sufficient to ex- plain the data ( H 0 ) or if g ( y, M χ ) ( H 1 ) provides a b etter fit. Although the issue of comparing non-nested models has been a ddressed since the early days of mo dern statistics ( Co x 1961 , 1962 , 2013 ; A tkinson 1970 ; Quand t 1974 ), as well as in the more recent physical literature ( Pilla, Loader & T a ylor 2005 ; Pilla & Loader 2006 ), a method with the desired sta- tistical prop erties, easy implementation and computational efficiency in astrophysics is still lacking. This article is arranged as follows. Section 2 reviews the LR T, Wilks’s theorem and their extensions to non-regular situations. Our prop osal for testing non-tested mo dels is in- troduced in Section 3 , v alidated via simulation studies in Section 4 , and applied to a realistic simulation of the F ermi- LA T γ -ray satellite in Section 5 . General discussion app ears in Section 6 . 2 WILKS, CHERNOFF AND TRIAL F A CTORS Let f ( y ; α ) and g ( y , β ) b e pdfs of the background and signal, where y is the detected energy , α and β are parameters. Suppose observed particles are a mixture of background and source, i.e., (1 − η ) f ( y , α ) + η g ( y , β ) (1) where 0 6 η 6 1 is the proportion of signal count s. A h yp othesis test can b e sp ecified as H 0 : η = η 0 v ersus H 1 : η > η 0 , and if β is kno wn the LR T statistic by T ( β ) = − 2 log L ( η 0 , ˆ α 0 , -) L ( ˆ η 1 , ˆ α 1 , β ) , (2) where L ( η , α , β ) is the likelihoo d function under ( 1 ). The nu- merator and denominator of ( 2 ) are the maximum likel iho od ac hiev able under H 0 and H 1 , resp ectiv ely with ˆ α 0 being the MLE of α under H 0 and ˆ α 1 and ˆ η 1 the MLEs under H 1 . ( Wilks 1938 ) states that when H 0 is true and when testing for a one-dimensional parameter (in this case η ), T ( β ) is asymptotically distributed as a χ 2 1 (the subscript being the degrees of freedom). Among the regularity conditions which guaran tee this result are: R C1. The mo dels are nested, meaning that there is a full model of which b oth H 0 and H 1 are special cases. R C2. The set of p ossible parameters of H 0 is on the inte- rior of that for the full mo del. R C3. The full mo del is identifiable under H 0 . Unfortunately in practice, it is common to encoun ter non- regular problems. Notice for example, if β is known b ut η 0 = 0, R C2 do es not hold. In this case, Chernoff ( 1954 ) applies; it generalizes Wilks and states that if H 0 is on the boundary of the parameter space, the asymptotic distribution of T ( β ) is an equal mixture of a χ 2 1 and a Dirac delta function at 0, namely 1 2 χ 2 1 + 1 2 δ (0). F urther, if η 0 = 0 (on the b o undary) and β is un- kno wn, the mo del in ( 1 ) is not identifiable under H 0 and R C3 fails. This is kno wn in statistics as a test of hypoth- esis where a nuisance parameter is defined only under H 1 , or “trial correction” in astroph ysical literature. A solution based on theoretical result of Davies ( 1987 ) is proposed b y Gross & Vitells ( 2010 ). In particular, under H 0 , T ( β ) is a random pro cess indexed b y β , specifically if RC2 (but not R C3) holds { T ( β ) , β ∈ B } is asymptotically a χ 2 1 -process. A natural choice of test statistic is sup β T ( β ) and Gross & Vitells ( 2010 ) pro vides an approximation in the limit as c → ∞ for the tail probabilit y P (sup β T ( β ) > c ). Finally , if both R C2 and R C3 fail to hold (e.g., the imp ortan t case of η 0 = 0 with β unknown), w e sho w in our Supplemen- tary Material that because { T ( β ) , β ∈ B } is a 1 2 χ 2 1 + 1 2 δ (0) random pro cess, P (sup β T ( β ) > c ) ≈ P ( χ 2 1 > c ) 2 + E [ N ( c 0 ) | H 0 ] e − c − c 0 2 (3) where E [ N ( c 0 ) | H 0 ] is the expected nu mber of up crossings of the T ( β ) process ov er the threshold c 0 under H 0 and c 0 is chosen c 0 << c . (Details of how to choose c 0 are given in Gross & Vitells ( 2010 ), where ( 3 ) is also asserted, but without proof.) Although this approximation holds as c → ∞ , when c is small, the right hand side of ( 3 ) is an upp er bound for P (sup β T ( β ) > c ). Th us, basing inference on ( 3 ) is v alid, though p erhaps conserv ativ e. 3 ST A TISTICAL COMP ARISON OF NON-NESTED MODELS Suppose w e wish to compare tw o p dfs, f ( y , α ) and g ( y, β ), for which RC1 do es not apply , that is the tw o pdfs are not special cases of a full mo del and do not share a parameter space. Notice that in b oth f and g free parameters (i.e., α and β respectively) are presen t and t hus, the problem cannot be reduced to a test for simple h yp otheses as in Cousins ( 2005 ), see Co x ( 1961 ) for more details. W e require β to b e one dimensional and α to lie in the interior of its parameter space. The goal is to develop a test of the hypothesis: H 0 : f ( y , α ) versus H 1 : g ( y, β ) (4) Although f ( y , α ) and g ( y, β ) are non-nested, we can construct a comprehensive model which includes b oth as MNRAS 000 , 1 – 5 (2016) Non-neste d mo dels c omp arison 3 special cases. There are tw o reasonable formulation s. W e encoun tered the first in ( 1 ); the second is prop ortional to { f ( y , α ) } 1 − η { g ( y, β ) } η , with 0 6 η 6 1 in b oth formula- tions. As discussed in Cox ( 1962 , 2013 ); Atkinson ( 1970 ) and Quandt ( 1974 ), there are adv antages and disadv antages to b oth. F rom our p erspective, the additive form in ( 1 ) has the adv antage of more app ealing mathematical prop erties. Since no normalizing constant is in volv ed, the maximization of the log-like liho od reduces to numerical optimization. In con trast to the test discussed in Section 2 , the mo del in ( 1 ) is not viewed as a mixture of astroph ysical mo d els in whic h a certain prop ortion of ev ents, η , originates a pro cess rep- resen ted by one mo del, and the the remaining prop o rtion, 1 − η , originates from the completing pro cess represente d by the other model. Instead, ( 1 ) is a mathematical formaliza- tion used to em b ed the p dfs f ( y , α ) and g ( y, β ) and their corresponding parameters spaces into an ov erarching mo del via the auxiliary parameter η ( Quandt 1974 ). The ov erarch- ing mo del has not astrophysical interpretation, but helps us reform ulate the test in ( 4 ) into a suitable form, i.e., H 0 : η = 0 v ersus H 1 : η > 0 . (5) P erhaps a more natural formulation of ( 4 ) would be H 0 : η = 0 versus H 1 : η = 1. Unfortunately , neither Wilks’s or Chernoff ’s theorems apply to this form ulation since they rely on the asymptotic normalit y of the MLE under H 0 , whic h can only hold if there is a contin uum of possible v al- ues of η under H 1 , with η = 0 in its interior. With indirect dark matter detection, the formulation in ( 5 ) allows the al- ternativ e mo del to include b o th the case where dark mat- ter and known cosmic sources are present simoultaneously (0 < η < 1) and the case where only dark matter is present ( η = 1). In situations where intermediate v alues of η are not ph ysical we might, in addition to ( 5 ), test H 0 : η = 1 ver- sus H 1 : η < 1, i.e., in terchange the roles of the h yp otheses as discussed in Co x ( 1962 , 2013 ). In this case, the n uisance parameter α is required to b e one dimensional i.e., α = α . Under model ( 1 ), testing ( 5 ) is equiv alent to testing η on the b oundary with β only being defined under the alter- nativ e. W e can apply the metho ds discussed in Section 2 to solv e this problem. Notice that such methods can still b e applied if the tw o mo dels share additional parameters, γ , i.e., f ( y , γ , α ) and g ( y , γ , β ). How ever, the maximized lik e- lihoo ds in ( 2 ) must be replaced by their profile counterparts L (0 , ˆ γ 0 , ˆ α 0 ) and L ( η 1 , ˆ γ 1 , ˆ α 1 , β ) ( Davison 2003 ). 4 V ALID A TION ON DARK MA TTER MODELS W e illustrate the reliability of the method prop osed for test- ing non-nested mo dels using tw o sets of Monte Carlo sim- ulations. In T est 1, w e compare the t wo mo dels introduced in Section 1 with the aim of distinguishing b et ween a dark matter signal and a p o wer law distributed cosmic source. In T est 2, we make the same comparison but in the presence of pow er law distributed background. In this case, H 0 specifies as f ( y , δ, λ, E 0 , φ ) = (1 − λ ) δ E δ 0 k δ y δ +1 + λ φ k φ y φ +1 E φ 0 (6) and H 1 specifies g ( y, δ, λ, E 0 , M χ ) = (1 − λ ) δ E δ 0 k δ y δ +1 + λ e − 7 . 8 y M χ y 1 . 5 k M χ ; (7) 0 5 10 15 c log 10 ( p.values ) 1e−04 0.001 0.01 0.1 1 3 σ Monte Carlo GV approximation GV approximation omitting Chernoff P ( χ 2 1 > c ) 0.5P ( χ 2 1 > c ) 0 5 10 15 c log 10 ( p.values ) 1e−04 0.001 0.01 0.1 1 3 σ Monte Carlo GV approximation GV approximation omitting Chernoff P ( χ 2 1 > c ) 0.5P ( χ 2 1 > c ) Figure 1. Comparing the approximation in ( 3 ) (solid blue lines) with Monte Carlo estimation of P (sup T ( M χ ) > c ) (gra y dashed lines), for T est 1 (upp er panel) and T est 2 (lower panel). Ap- proximat ions correspondig to ( 3 ) without the Chernoff correction (blue dashed lines), a χ 2 approxim ation (ligh t blue dash-dotted lines) and a Chernoff-adjusted χ 2 approxim ation (light blue dot- ted lines) are also rep orted. Mon te Carlo p-v alues were obtained by simulating 10,000 datasets under H 0 , eac h of size 10,000 for both simulations. F or each simulated dataset sup M χ T ( M χ ) was computed o ver an M χ grid of size 100 for T est 1 and size 400 for T est 2. Mon te Carlo errors (gra y areas) w ere attained via error propagation ( Cow an 1998 ). where k φ , k δ and k M χ are the normalizing constan ts for each pdf, 0 < λ < 1, δ > 0, φ > 0, E 0 = 1, y ∈ [ E 0 , 100] and M χ ∈ [ E 0 , 100]. Note that in this case, the formulation in ( 1 ), with mixture parameter λ , is first used to sp ecify the signal existence ov er a (relativ ely w ell kno wn) backgrou nd, whilist in the next step, equation ( 1 ) is adopted as a merely mathematical to ol to treat the non-nested case (as described previously). F or simplicity , in T est 2, λ , the prop ortion of even ts MNRAS 000 , 1 – 5 (2016) 4 S. Algeri et al. M χ T ype I error 0 0.001 0.002 0.004 0.005 3 − σ 15 30 45 60 75 90 ● ● ● ● ● ● ● N=10 N=100 N=200 N=500 N=1000 M χ P ower ● ● ● ● ● ● ● N=10, DC N=100, DC N=200, DC N=500, DC 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 15 30 45 60 75 90 Figure 2. Simulated t yp e I errors (upper panel) and power func- tions (low er panel) for T est 1 with at 3 σ significance. Shaded areas indicate regions expected to contain 68% (dark gray) and 95% (light gray) of the symbols if the nominal type I error of 0.003 holds. F or b oth the type I error and p o wer curves 10000 Monte Carlo simulations were used. coming from dark matter, was fixed to 0.2. In b oth tests, we estimated the av erage num b er of uprcrossings E [ N ( c 0 ) | H 0 ] using 1,000 Monte Carlo simulati ons. Finally , the approx- imation to P (sup β T ( β ) > c ) is calculated using ( 3 ) on a grid of v alues of c . The results are compared with the re- spective Monte Carlo p-v alues in Figure 1 along with the χ 2 and Chernoff corrections one might compute ignoring the regularit y conditions in Section 2 . F or small c , the appro ximation in ( 3 ) is greater than its Mon te Carlo counterp art. As c increases, ho wev er, the appro ximation conv erges to the Monte Carlo estimates for a goo d approximation to the p-v alue, P (sup β T ( β ) > c ). The χ 2 and respective Chernoff-adjusted appro ximation lead to o ver optimistic p-v alues, whereas similar results to those at- tained with ( 3 ) are achiev ed when the factor of 2 that ac- coun ts for RC1 is omitted. This is not surprising since the righ t hand side of ( 3 ), is dominated by E [ N ( c 0 ) | H 0 ] (which also explains the wide discrepancy betw een ( 3 ) and the χ 2 appro ximations in Figure 1 ) and in practice, when testin g on the b oundary of the parameter space, E [ N ( c 0 ) | H 0 ] is typ- ically calculated simulating a 1 2 χ 2 1 + 1 2 δ (0) random process directly . Th us, the Chernoff correction is automatically im- plemen ted in the leading term of ( 3 ). It is not uncommon in practice, e.g. in astronomy , for the num b er of counts to be considerably smaller than the 10,000 used in Figure 1 . Th us, we conduct a sim ulation study to verify the type I error (i.e., the rate of false rejections of H 0 ) of the metho d with smaller samples and verify that the appro ximate p-v alue in ( 3 ) holds. The upp er panel of Fig- ure 2 rep orts the sim ulated type I errors with a detection threshold on the p-v alue of 0.003 (3 σ ) for different sample sizes when conducting T est 1. F or sample sizes of at least 100, the Monte Carlo results are consistent with the numer- ical 3 σ error rate. The low er panel of Figure 2 shows the pow er (probability of detection) curves at 3 σ of the same test for different sample sizes. F or all the v alues of M χ con- sidered, a sample size of 500 is sufficient to achiev e a p o wer of nearly 1. 5 APPLICA TION TO SIMULA TED DA T A FR OM THE FERMI-LA T The F ermi Large Area T elescop e (LA T) ( At woo d 2009 ) is a pair-con version γ -ray telescope on b oard the earth-orbiting F ermi satellite. It measures energies and images γ -ra ys b e- t ween ab out a 100 MeV and severa l T eV. One particular aspect is the γ -ray signal induced by dark matter annihi- lations, which gives rise to measurable signal from celestial ob jects, like the Milky W ay cent er or dwarf galaxies. Here w e apply the method prop osed in this letter to a dataset sim ulated with realistic representations of the effects of the detector and present backgrounds. W e considered a 5 years observ ation of putativ e dark matter source (dwarf galaxy- lik e) with dark matter annihilating into b-quark pairs and a mass of the dark matter particle of 35 GeV. This assumption is consisten t with the most generic and popular mo dels for dark matter, namely that it is in large part made of a W eakly In teracting Massiv e Particles (WIMP). It is also consisten t with recent claims of evidence for dark matter. The signal normalization corresponds to ab out 200 ev ents detected in the LA T. Roughly , this corresp onds to a dark matter source at the distance of the dw arf galaxy Seg ue1 (and with compa- rable dark matter densit y) and an annihilation cross-section of ∼ 2 · 10 − 25 cm 3 s − 1 ). W e find a 4 . 198 σ significance in fav or of the dark matter mo del. Scaling the even t rate down to 50 (i.e. considering a low er cross-section by a factor of 4 or lo wer density b y a factor of 16) we obtain 2 . 984 σ signifi- cance (result not sho wn). Adding complexity , w e introduce a background, for example γ -ra ys introduced b y our own Galaxy . W e then considered 2176 counts from a p o wer-la w distributed background source as in ( 6 )-( 7 ) and about 550 dark matter even ts. F or simplicity , the mixture parameter λ is fixed at 0.2. In this case, we find 2 . 9 σ significance in fa vor of the model in ( 7 ). As expected, introducing back- ground significant ly reduces the p o wer for distinguishing a dark matter source from a conv entio nal source. It should b e noted how ever that (unlike in a full analysis) we do not at- MNRAS 000 , 1 – 5 (2016) Non-neste d mo dels c omp arison 5 H 0 N ˆ η ˆ M χ sup LRT Sig. T est 1 η = 0 200 0.971 27 21.018 4 . 038 σ η = 1 200 p-v alue = 0 . 528 T est 2 η = 0 2726 0.999 30 12.096 2 . 673 σ η = 1 2726 p-v alue = 1 T able 1. Summary of the analysis on the F ermi LA T simulation comparing the mo dels in T ests 1 and 2. Estimates and Signifi- cances refer to the tests H 0 : η = 0 versus H 1 : η > 0. P-v alues refer to the tests H 0 : η = 1 versus H 1 : η < 1. tempt to reduce background by taking γ -ray directions into accoun t. 6 SUMMAR Y & DISCUSSION W e ha ve presen ted a tw o-step solution to a common prob- lem in experimental astrophysics: comparing comp eting non- nested mo dels. On the basis of the seminal work of Cox ( 1962 , 2013 ) and Atkinson ( 1970 ) the first step of our strat- egy requires the sp ecification of a comprehensiv e model whic h extends the parameter space of the models to b e com- pared. The problem of testing non-nested mo del is then re- duced to the lo ok-elsewhere effect, and thus the second step naturally recalls Gross & Vitells ( 2010 ) as an efficien t solu- tion to accurately approximate the significance of new sig- nals. The resulting procedure is easy to implemen t, do es not require extensive calculations on a case-by-case basis and is computationally more efficient than Monte Carlo simula- tions. Recen t dev elopments ( Algeri et al. 2016 ) in the nested case illustrate additional desirable statistical prop erties of Gross & Vitells ( 2010 ) with resp ect to Pilla, Loader & T ay- lor ( 2005 ) and Pilla & Loader ( 2006 ). Given the nature of the methodology prop osed in this letter, w e exp ect these finding to carry ov er to the non-nested case. An example of testing non-nested mo dels arises in the searc h for particle dark matter. W e use this example to v ali- date a nd illustrate the procedure. W e also demonstrate goo d performance in a realistic simulation of data that is collected with the F ermi-LA T γ -ray detector and used in the search for signal from dark matter annihilation. Although any pair of h yp othesized mo dels can b e ex- pressed as a sp ecial case of ( 1 ), this formulation alone do es not alwa ys provide a mechanism for a statistical hypothesis test. In the non-nested scenario analysed in this letter, v alid inference for the test in ( 5 ) can b e achiev ed by applying the methodology in Gross & Vitells ( 2010 ). This method, ho w- ev er, cannot handle multi-dimensi onal parameters that are defined only under H 1 , nor can it deal with n uisance param- eters under H 0 whic h lie on the b oundary of their parameter space. A p o ssible approach to tac kle the first limitation is to apply the theory in Vitells & Gross ( 2011 ) to the compre- hensiv e mo del in ( 1 ). Whereas an exten tion of the method to ov ercome the second limitation could rely on the the- ory in Self & Liang ( 1987 ). In light of this, the metho dology proposed is particularly suited to comparisons of non-nested models where these limitations often do not arise. Soft ware for the metho dology illuatsrated in this let- ter is av ailable at: http://wwwf.imperial.ac.uk/~sa2514/ Research.html . A CKNOWLEDGEMENTS The authors ackno wledge Brandon Anderson for using to ols publicly a v ailable from th e F ermi LA T Collaboration to sim- ulate F ermi LA T data. JC thanks the supp ort of the Knut and Alice W allenberg foundation and the Swedish Research Council. DvD ackno wledges supp ort from a W olfson Re- searc h Merit Award pro vided by the British Roy al So ciet y and from a Marie-Curie Career Int egration Gran t pro vided b y the Europ ean Commission. REFERENCES Ack ermann M. et al., 2011, Phys. Rev. Lett., 107, 241302 Ack ermann M. et al., 2012, Astrophys. J., 747, 121 Ack ermann M. et al., 2014, Phys. Rev. D, 89, 042001 Ack ermann M. et al., 2015, Phys. Rev. Lett., 115, 231301 Algeri S. et al., 2016, Submitted Atkinso n. A.C., 1970, JRSSB, 32, 3 At woo d W.B. et al., 2009, Astrophys. J., 697, 2 Bergstr ¨ om L., Ullio P ., Buckley J., 1998, Astropart. Ph ys., 9, 2 Chernoff. H., 1954, Ann. Math. Stat., 25, 3 Conrad. J., 2014, preprint Conrad J., Cohen-T anugi J., Stigari L.E., 2015, J. Exp. Theor. Phys., 148, 12 Cousins R. et al., 2005, J. High Energy Phys., 11, 046 Cow an G., 1998, Statistical Data Analysis, Oxford Univ. Press Inc., New Y ork Cox D.R., 1961, Pro ceedings of the F ourth Berk eley Symposium on Mathematical Statistics and Probability , 1, p. 105 Cox. D.R., 1962, J. R. Stat. So c. B, 24, 2 Cox. D.R., 2013, J. R. Stat. So c. B, 75, 2 Davies R.B., 1987, Biometrik a, 74, 1 Daylan T. et al., 2014, preprint Davison A.C., 2003, Statistical Models, Cambridge Univ. Press Geringer-Sameth A., Koushiappas S.M., 2011, Ph ys. Rev. Lett. 107, 241303 Geringer-Sameth A. et al., 2015, Phys. Rev. Lett., 115, 081101 Gross E., Vitells O., 2010, Eur. Phys. J. C, 70, 1-2 Guiriec S. et al., 2015, Astrophys. J., 807, 2 Lyons L., 2008, Ann. Appl. Stat., 2, 3 Pilla R., Loader C., T aylor C.C., 2005, Phys. Rev. Lett., 95, 230202 Pilla R., Loader C., 2006, preprint (arXiv:math/0511503v2) Profumo S., Linden T., 2012, preprint Quandt R.E., 1974, Rev. Econ. Stat., 5, 1 Self S.G., Liang K.-Y. 1987, JASA, 82, 398 Vitells O., Gross E., 2011, Astropart. Phys., 35, 5 Wilks S.S., 1938, Ann. Math. Stat., 9 This pap er has b een t yp eset from a T E X/L A T E X file prepared by the author. MNRAS 000 , 1 – 5 (2016)