Innovated higher criticism for detecting sparse signals in correlated noise

Higher criticism is a method for detecting signals that are both sparse and weak. Although first proposed in cases where the noise variables are independent, higher criticism also has reasonable performance in settings where those variables are correlated. In this paper we show that, by exploiting the nature of the correlation, performance can be improved by using a modified approach which exploits the potential advantages that correlation has to offer. Indeed, it turns out that the case of independent noise is the most difficult of all, from a statistical viewpoint, and that more accurate signal detection (for a given level of signal sparsity and strength) can be obtained when correlation is present. We characterize the advantages of correlation by showing how to incorporate them into the definition of an optimal detection boundary. The boundary has particularly attractive properties when correlation decays at a polynomial rate or the correlation matrix is Toeplitz.

💡 Research Summary

Higher criticism (HC) has become a cornerstone for detecting signals that are simultaneously sparse and weak in massive multiple‑testing problems. Classical HC theory, however, rests on the assumption that the noise variables are independent. In many modern applications—genomics, neuroimaging, finance, and sensor networks—observations exhibit substantial correlation, and the impact of such dependence on HC performance has remained largely unexplored. This paper overturns the conventional wisdom that correlation necessarily degrades detection power. By explicitly modeling and exploiting the covariance structure, the authors develop a “correlation‑adjusted” higher criticism statistic that achieves strictly better detection thresholds than the original HC in a wide range of realistic settings.

Model and Notation
The authors consider an n‑dimensional Gaussian vector
(Z = \mu + \varepsilon,\quad \varepsilon \sim N(0,\Sigma)),
where the mean vector (\mu) is s‑sparse (only s components are non‑zero) and each non‑zero entry equals a common signal strength (\tau). The covariance matrix (\Sigma) is assumed to be positive definite and to possess a structured form: either Toeplitz ((\Sigma_{ij}= \rho_{|i-j|})) or a polynomially decaying correlation ((|\rho_k| \le C k^{-\alpha}) for some (\alpha>0)).

From Classical to Correlation‑Adjusted HC
Classical HC computes p‑values under the null (N(0,1)) and forms
(\mathrm{HC}n = \max{1\le i\le n}\sqrt{n},\frac{i/n-p_{(i)}}{\sqrt{p_{(i)}(1-p_{(i)})}}).
When the noise is correlated, the distribution of each component of (Z) is no longer standard normal, and the dependence inflates the variance of the empirical p‑value process. The authors propose to pre‑whiten the data by applying (\Sigma^{-1/2}): (Y = \Sigma^{-1/2} Z). Although the components of (Y) remain correlated, the transformation normalizes the marginal variances and makes the dependence structure explicit in the covariance of the resulting p‑values.

The new statistic is defined as
\