The Marginal Likelihood of two-way tables and Ecological Inference

The paper derives new results on the marginal likelihood of a two-way table which clarify the conditions under which Ecological inference is possible and lead to an efficient algorithm for maximizing the exact multinomial likelihood. The first part generalizes the work of Placket(1977} on the marginal likelihood of a 2 x 2 table to a general R x C table. In doing so, new conceptual tools are introduced and new insights on the geometry of the collection of tables having fixed row and column margins and the extended hypergeometric distribution are derived. In the second part, when observations on the row and the column marginal distributions are available for a collection of two-way tables sharing the same association structure, an efficient Fisher scoring algorithm for maximizing the exact likelihood under multinomial sampling is introduced and a small simulation study is used to compare the performance of the proposed method with two well established ones.

💡 Research Summary

The manuscript “The Marginal Likelihood of Two‑Way Tables and Ecological Inference” tackles two intertwined problems that lie at the heart of ecological inference: (i) the theoretical limits of inference from a single contingency table when only row and column margins are observed, and (ii) the practical estimation of conditional row probabilities when many tables share the same association structure.

In the first part the authors extend Plackett’s (1977) analysis of the 2 × 2 case to a general R × C table. They formalize the set 𝒩 of all possible count tables compatible with given positive row and column totals and express the exact multinomial marginal likelihood as a sum over 𝒩 weighted by an “extended hypergeometric” factor. By re‑parameterizing the conditional row probabilities p_{j|i} with logits φ_j and log‑odds ratios λ_{ij}, the log‑likelihood decomposes into a term involving the margins and a combinatorial sum of exp(V(N,Λ)+γ(N)). The likelihood equations reduce to two simple constraints: (2) the column totals must equal the weighted sum of conditional probabilities, and (3) the expected cell counts under the extended hypergeometric distribution must match the observed counts for interior cells.

Crucially, the authors prove that no interior table in 𝒩 can satisfy (3) exactly; instead, a special family of “extreme tables” Z—constructed by permuting rows and columns and then greedily allocating mass to the largest feasible cells—approaches the likelihood maximum. By perturbing zero cells with a small ε and scaling rows, they obtain a sequence Z(ε) whose log‑odds ratios diverge (order O(−log ε) or O(−2 log ε)). As ε → 0, the corresponding conditional probability matrix P(Z,ε) satisfies the likelihood equations in the limit, establishing that each extreme table defines a local maximum of the marginal likelihood. These maxima correspond to the Fréchet class’s upper bound, i.e., the table with the strongest possible positive association given the margins. Consequently, the maximum‑likelihood estimate for a single table is inconsistent for the true conditional probabilities; the data cannot identify a unique p_{j|i}.

The second part shifts focus to a collection of tables {T_k} that share the same conditional row distribution (the ecological inference setting). Here the exact marginal likelihood is the product of the individual likelihoods, each still requiring a sum over its own 𝒩_k. The authors propose to pre‑compute and store each 𝒩_k (feasible because the margins are fixed) and then evaluate the extended hypergeometric expectations efficiently. They derive the score vector and Fisher information matrix for the joint likelihood and present a Fisher‑scoring algorithm that iteratively updates the log‑odds parameters. A necessary condition for the information matrix to be nonsingular is given, ensuring that the algorithm is well‑behaved.

A modest simulation study compares the new Fisher‑scoring estimator with two established ecological inference methods: (a) a naïve independence‑based estimator that uses column margins only, and (b) an EM‑based Bayesian hierarchical approach. Across a range of R × C configurations, sample sizes, and margin imbalances, the proposed method consistently yields lower bias and mean‑squared error, while computational time is comparable once the feasible‑table libraries are built. The authors also illustrate the pathological behavior of the single‑table likelihood (a saddle point at independence) and show that the extreme‑table likelihoods dominate.

Overall, the paper makes three substantive contributions:

1. Theoretical Clarification – It rigorously characterizes the geometry of the set of tables with fixed margins, identifies the Fréchet‑class upper bound as the source of local likelihood maxima, and demonstrates the impossibility of consistent inference from a single table.

2. Algorithmic Innovation – It introduces an exact, Fisher‑scoring based maximization routine for the joint multinomial likelihood of multiple tables sharing a common conditional structure, handling the combinatorial complexity through pre‑computed feasible‑table sets.

3. Empirical Validation – It provides simulation evidence that the new estimator outperforms standard ecological inference techniques, especially in small‑sample or highly unbalanced margin scenarios.

The manuscript is well‑structured, the proofs are mathematically sound, and the algorithmic details are sufficiently explicit for replication. Minor improvements could include a discussion of scalability to very large R or C (e.g., using Monte‑Carlo approximations for the sum over 𝒩) and a more extensive comparison with recent Bayesian non‑parametric ecological inference methods. Nonetheless, the work represents a significant advance in both the theory and practice of ecological inference.