Overall and Pairwise Segregation Tests Based on Nearest Neighbor Contingency Tables

Multivariate interaction between two or more classes (or species) has important consequences in many fields and causes multivariate clustering patterns such as segregation or association. The spatial segregation occurs when members of a class tend to be found near members of the same class (i.e., near conspecifics) while spatial association occurs when members of a class tend to be found near members of the other class or classes. These patterns can be studied using a nearest neighbor contingency table (NNCT). The null hypothesis is randomness in the nearest neighbor (NN) structure, which may result from – among other patterns – random labeling (RL) or complete spatial randomness (CSR) of points from two or more classes (which is called the CSR independence, henceforth). In this article, we introduce new versions of overall and cell-specific tests based on NNCTs (i.e., NNCT-tests) and compare them with Dixon’s overall and cell-specific tests. These NNCT-tests provide information on the spatial interaction between the classes at small scales (i.e., around the average NN distances between the points). Overall tests are used to detect any deviation from the null case, while the cell-specific tests are post hoc pairwise spatial interaction tests that are applied when the overall test yields a significant result. We analyze the distributional properties of these tests; assess the finite sample performance of the tests by an extensive Monte Carlo simulation study. Furthermore, we show that the new NNCT-tests have better performance in terms of Type I error and power. We also illustrate these NNCT-tests on two real life data sets.

💡 Research Summary

The paper addresses the problem of detecting and characterizing spatial interaction between two or more classes (or species) using nearest‑neighbor contingency tables (NNCTs). While Dixon’s overall and cell‑specific NNCT tests have been the standard tools, they rely on the assumption that row and column totals of the table are fixed, which limits their accuracy especially when class sizes are unequal or sample sizes are small. The authors therefore propose new formulations for both overall and pairwise (cell‑specific) tests that are valid under the two common null models: random labeling (RL) and complete spatial randomness (CSR) independence.

The methodological contribution begins with a rigorous derivation of the expected cell counts and their variances under RL and CSR. Unlike Dixon’s approach, which approximates the variance by treating each cell as independent, the new derivations incorporate the covariance structure induced by the NN relationship, yielding more accurate variance estimates. The overall test statistic is defined as a weighted sum of squared deviations,

\