Fourier transform inequalities for phylogenetic trees

Phylogenetic invariants are not the only constraints on site-pattern frequency vectors for phylogenetic trees. A mutation matrix, by its definition, is the exponential of a matrix with non-negative off-diagonal entries; this positivity requirement implies non-trivial constraints on the site-pattern frequency vectors. We call these additional constraints edge-parameter inequalities.'' In this paper, we first motivate the edge-parameter inequalities by considering a pathological site-pattern frequency vector corresponding to a quartet tree with a negative internal edge. This site-pattern frequency vector nevertheless satisfies all of the constraints described up to now in the literature. We next describe two complete sets of edge-parameter inequalities for the group-based models; these constraints are square-free monomial inequalities in the Fourier transformed coordinates. These inequalities, along with the phylogenetic invariants, form a complete description of the set of site-pattern frequency vectors corresponding to \emph{bona fide} trees. Said in mathematical language, this paper explicitly presents two finite lists of inequalities in Fourier coordinates of the form monomial $\leq 1$,’’ each list characterizing the phylogenetically relevant semialgebraic subsets of the phylogenetic varieties.

💡 Research Summary

The paper addresses a fundamental gap in the mathematical description of phylogenetic models: while phylogenetic invariants (polynomial equations that vanish on the model variety) capture many constraints on site‑pattern frequency vectors, they do not encode the positivity inherent in the definition of a mutation matrix. By definition, a mutation matrix is the exponential of a rate matrix whose off‑diagonal entries are non‑negative. This structural requirement imposes additional, non‑trivial constraints on the admissible frequency vectors, which the authors term “edge‑parameter inequalities.”

To illustrate the insufficiency of invariants alone, the authors construct a pathological example: a quartet tree with a negative internal edge length. The corresponding site‑pattern frequency vector satisfies every known invariant for the chosen group‑based model, yet it cannot arise from any legitimate stochastic process because the underlying rate matrix would contain a negative off‑diagonal entry. This example motivates the search for extra constraints that rule out such impossible configurations.

The core technical contribution is the derivation of two complete families of edge‑parameter inequalities for all group‑based models (including Jukes‑Cantor, Kimura 2‑parameter, and Kimura 3‑parameter). The authors first apply the discrete Fourier transform (also known as the Hadamard or character transform) to the site‑pattern frequencies. In Fourier space, each edge of the tree is associated with a single scalar parameter—essentially the eigenvalue of the transition matrix on that edge. Because the transition matrix is an exponential of a non‑negative rate matrix, each Fourier parameter must lie in the interval