Rational subsets of groups

This text, Chapter 23 in the “AutoMathA” handbook, is devoted to the study of rational subsets of groups, with particular emphasis on the automata-theoretic approach to finitely generated subgroups of free groups. Indeed, Stallings’ construction, associating a finite inverse automaton with every such subgroup, inaugurated a complete rewriting of free group algorithmics, with connections to other fields such as topology or dynamics. Another important vector in the chapter is the fundamental Benois’ Theorem, characterizing rational subsets of free groups. The theorem and its consequences really explain why language theory can be successfully applied to the study of free groups. Rational subsets of (free) groups can play a major role in proving statements (a priori unrelated to the notion of rationality) by induction. The chapter also includes related results for more general classes of groups, such as virtually free groups or graph groups.

💡 Research Summary

The paper “Rational subsets of groups” provides a comprehensive survey of rational subsets in groups, with a strong emphasis on the automata‑theoretic approach to finitely generated subgroups of free groups. It is divided into three main sections.

The first section introduces basic notions concerning finitely generated groups, focusing on free groups. After recalling the construction of a free group (F_A) from an involutive alphabet (eA = A \cup A^{-1}) and the associated reduction system, the authors list classical decision problems such as the word problem, the conjugacy problem, the generalized word problem, the order problem, and the isomorphism problem. They point out that in a free group the word problem is solved by simple reduction, and they define the length of an element as the length of its reduced word.

The second section is devoted to inverse automata and Stallings’ folding construction. An inverse automaton is defined as a deterministic, trim automaton over an involutive alphabet that has a single final state (the basepoint) and contains for each edge ((p,a,q)) the inverse edge ((q,a^{-1},p)). The authors prove that any language recognized by an inverse automaton can be described by a morphism to another inverse automaton, establishing a minimality property.

Stallings’ construction starts from a finite set of reduced words (X\subseteq R_A). One builds the “flower automaton” (F(X)) by attaching a petal for each word, then repeatedly applies Stallings foldings: whenever two distinct edges with the same label emanate from the same vertex, they are identified together with their inverses. The process terminates in a finite inverse automaton (S(X)), called the Stallings automaton of the subgroup (H=\langle X\rangle). Crucially, (S(X)) is independent of the chosen generating set and of the folding order; it uniquely represents the subgroup.

Using this representation, the generalized word problem in a free group becomes trivial: a word (u) belongs to (H) iff the reduced word of (u) labels a loop at the basepoint in (S(H)). The algorithm runs in time (O(n\log^{*}n+m)), where (n) is the total length of the generators and (m) is the length of the input word. The authors also show how to extract a free basis of (H) from a spanning tree of (S(H)); the edges not belonging to the tree correspond to basis elements, giving an algorithmic proof of the Nielsen‑Schreier theorem for finitely generated subgroups.

The third section broadens the discussion to rational and recognizable subsets of arbitrary groups. A rational subset of a group (G) is defined as the image under the canonical projection of a regular language over the generating alphabet. Recognizable subsets arise from finite index subgroups via the Nerode equivalence. The central result is Benois’ Theorem, which states that in a free group the class of rational subsets coincides with the class of images of regular languages; equivalently, every rational subset of a free group is a regular language after applying the natural projection. This theorem explains why language‑theoretic techniques are effective in free group theory.

Consequences of Benois’ theorem include closure of rational subsets under product, inverse, union, and intersection, as well as the fact that solution sets of equations with rational constraints are again rational. The authors then discuss extensions to more general classes of groups, such as virtually free groups and right‑angled Artin (graph) groups. In virtually free groups the automatic structure guarantees decidability of the generalized word problem and effective manipulation of rational subsets. For graph groups, certain subclasses retain closure properties, while others require more delicate analysis.

The paper also touches on dynamical aspects: rational subsets can be used to describe invariant sets of group actions, cellular automata, and other dynamical systems. The authors note that inverse automata provide a concrete visual tool for studying such dynamics, for example by representing fixed points or periodic points as loops in the automaton.

Finally, the authors comment on algorithmic implementations. The most efficient known folding algorithm is due to Touikan, with complexity (O(n\log^{*}n)). Existing software packages such as CRAG, GAP’s AUTOMATA and FGA modules already implement Stallings’ construction and related algorithms, making the theory readily applicable in computational group theory.

In summary, the chapter offers a unified view of rational subsets across free groups and their generalizations, showing how Stallings automata give a canonical, algorithmically tractable representation of finitely generated subgroups, and how Benois’ theorem bridges group theory with formal language theory. These tools not only solve classical decision problems efficiently but also open pathways to applications in topology, dynamics, and the study of more complex group families.