On the Use of Nearest Neighbor Contingency Tables for Testing Spatial Segregation

For two or more classes (or types) of points, nearest neighbor contingency tables (NNCTs) are constructed using nearest neighbor (NN) frequencies and are used in testing spatial segregation of the classes. Pielou’s test of independence, Dixon’s cell-specific, class-specific, and overall tests are the tests based on NNCTs (i.e., they are NNCT-tests). These tests are designed and intended for use under the null pattern of random labeling (RL) of completely mapped data. However, it has been shown that Pielou’s test is not appropriate for testing segregation against the RL pattern while Dixon’s tests are. In this article, we compare Pielou’s and Dixon’s NNCT-tests; introduce the one-sided versions of Pielou’s test; extend the use of NNCT-tests for testing complete spatial randomness (CSR) of points from two or more classes (which is called \emph{CSR independence}, henceforth). We assess the finite sample performance of the tests by an extensive Monte Carlo simulation study and demonstrate that Dixon’s tests are also appropriate for testing CSR independence; but Pielou’s test and the corresponding one-sided versions are liberal for testing CSR independence or RL. Furthermore, we show that Pielou’s tests are only appropriate when the NNCT is based on a random sample of (base, NN) pairs. We also prove the consistency of the tests under their appropriate null hypotheses. Moreover, we investigate the edge (or boundary) effects on the NNCT-tests and compare the buffer zone and toroidal edge correction methods for these tests. We illustrate the tests on a real life and an artificial data set.

💡 Research Summary

This paper provides a comprehensive re‑examination of nearest‑neighbor contingency tables (NNCTs) as tools for testing spatial segregation among two or more classes of points. An NNCT records, for each point taken as a “base,” the class of its nearest neighbor (NN) and thus forms a cross‑tabulation of base‑NN class frequencies. The authors focus on four NNCT‑based tests: Pielou’s test of independence, and Dixon’s cell‑specific, class‑specific, and overall tests. While all of these methods were originally derived under the null hypothesis of random labeling (RL) for completely mapped data, the paper demonstrates that only Dixon’s tests are theoretically appropriate for RL; Pielou’s test, despite its popularity, fails to respect the dependence structure inherent in fully observed (base, NN) pairs.

The study expands the scope of NNCT testing to the null hypothesis of complete spatial randomness (CSR) independence, where each class is generated by an independent homogeneous Poisson process. The authors derive one‑sided versions of Pielou’s test, prove that they are also liberal under both RL and CSR, and show that Pielou’s test is only valid when the NNCT is built from a random sample of (base, NN) pairs rather than from the full data set.

A large‑scale Monte Carlo simulation forms the empirical backbone of the paper. The authors vary point intensity, class proportion, and total sample size across thousands of replicates, estimating empirical significance levels and power for each test under both RL and CSR. Results consistently reveal that Dixon’s overall and cell‑specific tests maintain the nominal α = 0.05 level across all scenarios, whereas Pielou’s test (both two‑sided and one‑sided) exhibits inflated type‑I error rates, especially with small samples or unbalanced class sizes. In terms of power, Dixon’s tests dominate, detecting segregation reliably under a range of alternative patterns; Pielou’s test only matches Dixon’s performance in a narrow set of strong‑segregation cases.

The paper also addresses edge effects, a common source of bias in spatial analyses. Two correction strategies are compared: a buffer‑zone approach that discards (base, NN) pairs near the study‑area boundary, and a toroidal (wrap‑around) correction that treats the study region as a torus. Simulations show that both methods improve the adherence of test statistics to their asymptotic χ² distributions, but the toroidal correction is computationally simpler and more generally applicable.

Real‑world applicability is illustrated with two data sets. The first is an ecological data set of two tree species, where Dixon’s overall test yields a significant p‑value (≈0.02) indicating genuine segregation, while Pielou’s test produces an overly small p‑value (≈0.004), suggesting a false sense of stronger segregation. The second is an artificial data set generated under CSR and under a strong segregation alternative; Dixon’s tests correctly maintain the 0.05 level under CSR and achieve high power under segregation, whereas Pielou’s test remains liberal under CSR (empirical size ≈0.12).

The authors prove the consistency of Dixon’s tests under both RL and CSR: as the number of points grows, the test statistics converge in distribution to the χ² limit, guaranteeing asymptotic validity. By contrast, Pielou’s test lacks this property unless the NNCT is based on a truly random sample of (base, NN) pairs, a condition rarely met in practice.

In conclusion, the paper recommends the exclusive use of Dixon’s NNCT‑based tests for spatial segregation analysis, especially when the data are completely mapped. It advises applying toroidal edge correction to mitigate boundary bias and warns against the use of Pielou’s test (including its one‑sided variants) for RL or CSR hypotheses. The extensive simulation evidence, theoretical proofs, and illustrative examples together provide a solid methodological foundation for practitioners seeking reliable inference on spatial interaction patterns.