Statistical inference optimized with respect to the observed sample for single or multiple comparisons

The normalized maximum likelihood (NML) is a recent penalized likelihood that has properties that justify defining the amount of discrimination information (DI) in the data supporting an alternative hypothesis over a null hypothesis as the logarithm of an NML ratio, namely, the alternative hypothesis NML divided by the null hypothesis NML. The resulting DI, like the Bayes factor but unlike the p-value, measures the strength of evidence for an alternative hypothesis over a null hypothesis such that the probability of misleading evidence vanishes asymptotically under weak regularity conditions and such that evidence can support a simple null hypothesis. Unlike the Bayes factor, the DI does not require a prior distribution and is minimax optimal in a sense that does not involve averaging over outcomes that did not occur. Replacing a (possibly pseudo-) likelihood function with its weighted counterpart extends the scope of the DI to models for which the unweighted NML is undefined. The likelihood weights leverage side information, either in data associated with comparisons other than the comparison at hand or in the parameter value of a simple null hypothesis. Two case studies, one involving multiple populations and the other involving multiple biological features, indicate that the DI is robust to the type of side information used when that information is assigned the weight of a single observation. Such robustness suggests that very little adjustment for multiple comparisons is warranted if the sample size is at least moderate.

💡 Research Summary

The paper introduces a novel evidence measure called Discrimination Information (DI) that is built on the Normalized Maximum Likelihood (NML) ratio. NML is a penalized likelihood that normalizes the maximized likelihood over the entire sample space, thereby automatically incorporating a complexity penalty for the model. DI is defined as the logarithm of the NML ratio between an alternative hypothesis and a null hypothesis. In this way, DI plays a role analogous to the Bayes factor—quantifying the strength of evidence in favor of the alternative—while avoiding the need for a prior distribution.

The authors prove that under weak regularity conditions, the probability of “misleading evidence” (evidence that favours the wrong hypothesis) converges to zero as the sample size grows. This asymptotic property mirrors the consistency of Bayes factors but is achieved without averaging over outcomes that did not actually occur; DI is minimax optimal with respect to the observed sample alone. Moreover, DI can be applied to simple point null hypotheses (e.g., θ = θ₀) without requiring any special treatment, a feature that Bayes factors lack when the prior on the null is improper or undefined.

A practical obstacle is that NML is undefined for many models, especially those with continuous parameter spaces where the normalizing constant diverges. To overcome this, the authors propose a weighted‑likelihood extension. By assigning a weight to the likelihood function—derived either from side information associated with other comparisons or from knowledge about the null parameter value—the weighted NML becomes finite, and DI can be computed. The weight is interpreted as the contribution of an “extra observation” of side information.

Two empirical case studies illustrate the methodology. The first involves multiple population means, where each group is compared against a common control. The second examines many biological features (e.g., gene expression levels) across conditions. In both settings, the authors assign a weight equivalent to a single observation to the side information. The resulting DI values are remarkably stable across different choices of side information, indicating robustness. Importantly, when the sample size is moderate to large, the DI‑based inference requires little or no adjustment for multiple comparisons, contrasting sharply with traditional corrections such as Bonferroni or false‑discovery‑rate procedures that can dramatically reduce power.

The paper’s contributions can be summarized as follows:

1. A prior‑free evidence metric – DI provides a log‑scale measure of evidence comparable to Bayes factors but without the subjectivity of prior specification.
2. Minimax optimality for the observed data – The method optimizes performance with respect to the realized sample, not an average over hypothetical samples, guaranteeing asymptotic vanishing of misleading evidence.
3. Extension to otherwise intractable models – Weighted likelihoods render NML finite for models where the unweighted version is undefined, broadening the applicability of DI.
4. Empirical robustness and reduced multiple‑testing burden – Case studies show that DI is insensitive to the exact form of side information when the weight is modest, and that moderate sample sizes obviate heavy multiple‑testing corrections.

The authors conclude by outlining future research directions: systematic design of optimal weights, extensions to non‑i.i.d. or streaming data contexts, and deeper theoretical exploration of finite‑sample guarantees. If these avenues are pursued, DI could become a cornerstone of statistical inference, offering a principled, data‑centric alternative to p‑values and Bayes factors, especially in high‑dimensional scientific domains such as genomics, neuroimaging, and large‑scale clinical trials.