On Recognizable Tree Languages Beyond the Borel Hierarchy

We investigate the topological complexity of non Borel recognizable tree languages with regard to the difference hierarchy of analytic sets. We show that, for each integer $n \geq 1$, there is a $D_{\omega^n}({\bf \Sigma}^1_1)$-complete tree language L_n accepted by a (non deterministic) Muller tree automaton. On the other hand, we prove that a tree language accepted by an unambiguous B"uchi tree automaton must be Borel. Then we consider the game tree languages $W_{(i,k)}$, for Mostowski-Rabin indices $(i, k)$. We prove that the $D_{\omega^n}({\bf \Sigma}^1_1)$-complete tree languages L_n are Wadge reducible to the game tree language $W_{(i, k)}$ for $k-i \geq 2$. In particular these languages $W_{(i, k)}$ are not in any class $D_{\alpha}({\bf \Sigma}^1_1)$ for $\alpha < \omega^\omega$.

💡 Research Summary

The paper investigates the topological complexity of recognizable tree languages beyond the Borel hierarchy by employing the difference hierarchy of analytic sets, denoted (D_{\alpha}(\mathbf{\Sigma}^1_1)). After a concise introduction to infinite binary trees, Muller and Büchi tree automata, and basic descriptive set theory, the authors present three main contributions.

First, for every integer (n \ge 1) they construct a tree language (L_n) that is (D_{\omega^n}(\mathbf{\Sigma}^1_1))-complete. Each (L_n) is accepted by a nondeterministic Muller tree automaton. The construction encodes (\omega^n) iterations of the analytic difference operation into the acceptance condition of the automaton: the set of colors seen infinitely often along a branch must belong to a prescribed (\omega^n)-level difference of (\mathbf{\Sigma}^1_1) sets. Completeness is proved by showing that any set in (D_{\omega^n}(\mathbf{\Sigma}^1_1)) can be continuously reduced to (L_n). This demonstrates that Muller automata can recognize languages situated arbitrarily high in the analytic difference hierarchy.

Second, the authors turn to unambiguous Büchi tree automata. They prove that any language accepted by such an automaton is necessarily Borel. The proof relies on the fact that Büchi acceptance can be expressed as a countable union of closed conditions, which forces the recognized set to lie within the Borel sigma‑algebra. Consequently, unambiguous Büchi automata cannot capture any of the non‑Borel languages constructed in the first part, highlighting a sharp separation between the expressive powers of Büchi and Muller models.

Third, the paper studies the game tree languages (W_{(i,k)}) associated with Mostowski–Rabin indices ((i,k)). These languages arise from infinite two‑player games on trees where the winner is determined by the parity of the minimal index appearing infinitely often. For indices satisfying (k-i \ge 2), the authors show that each (L_n) is Wadge‑reducible to (W_{(i,k)}); that is, there exists a continuous function (f) such that (L_n = f^{-1}(W_{(i,k)})). This reduction implies that (W_{(i,k)}) lies strictly above all (D_{\omega^n}(\mathbf{\Sigma}^1_1)) levels. Moreover, they prove that (W_{(i,k)}) does not belong to any class (D_{\alpha}(\mathbf{\Sigma}^1_1)) for (\alpha < \omega^{\omega}). Hence these game languages occupy a very high position in the analytic difference hierarchy, serving as natural examples of tree languages of extreme topological complexity.

The paper concludes by discussing the implications of these findings for the theory of infinite tree automata, descriptive set theory, and the classification of regular tree languages. It suggests further research directions, such as pinpointing exact Wadge degrees of other regular tree languages, exploring the impact of determinism versus nondeterminism on the difference hierarchy, and extending the analysis to other acceptance conditions like parity or Rabin. Overall, the work provides a comprehensive map of how recognizable tree languages populate the landscape beyond Borel sets, establishing clear boundaries between different automata models and enriching our understanding of the interplay between automata theory and descriptive set theory.