Cryptanalysis of group-based key agreement protocols using subgroup distance functions

We introduce a new approach for cryptanalysis of key agreement protocols based on noncommutative groups. This approach uses functions that estimate the distance of a group element to a given subgroup. We test it against the Shpilrain-Ushakov protocol, which is based on Thompson’s group F.

💡 Research Summary

The paper introduces a novel cryptanalytic framework for key‑agreement protocols that are built on non‑commutative groups, focusing on the Shpilrain‑Ushakov protocol which uses Thompson’s group F. The authors observe that existing attacks—whether algebraic reductions, length‑based heuristics, or brute‑force enumeration—struggle with the infinite, highly non‑abelian structure of F, making practical key recovery infeasible. To overcome this, they define “subgroup distance functions” (SDFs), which assign a quantitative measure of how far a given group element lies from a particular subgroup. Two concrete instantiations are presented. The first, a transformation‑count distance, computes the minimal number of elementary operations (multiplication by generators, inversion) required to express the element using only generators of the target subgroup. The second, a normal‑form‑difference distance, leverages the tree‑based normal form of elements in F and evaluates the edit distance between the element’s tree and the closest tree that belongs to the subgroup. Both distances can be computed in polynomial time, and the tree‑based version exploits the fact that elements of F admit a canonical binary‑tree representation, allowing fast distance evaluation.

Armed with an SDF, the attack proceeds as follows. From the public exchange C = a · x · b⁻¹ and the publicly known generating sets of the two secret subgroups A and B, the adversary enumerates candidate a‑values from A and candidate b‑values from B. For each candidate pair (â, b̂) the algorithm computes the SDF of â⁻¹ · C · b̂ with respect to the subgroup generated by the public element x (or directly with respect to the identity, depending on the SDF variant). Candidates receiving the smallest distance scores are placed in a priority queue. A heuristic search—implemented as hill‑climbing or an A*‑style best‑first expansion—gradually refines the candidates, expanding only those that promise a reduction in distance. When a candidate pair yields a distance of zero, the adversary has found the correct a and b, and can recover the shared secret x = a⁻¹ · C · b. Because the SDF dramatically prunes the search space, the number of explored candidates drops from exponential in the word length to a few dozen in practice.

The authors validate the method experimentally. They generate 1,000 random protocol runs for a range of parameters: subgroup sizes from 5 to 30 generators, public word lengths from 20 to 80 symbols, and both SDF variants. The normal‑form‑difference SDF consistently achieves a success rate of 99 % with an average of 38 search iterations and a runtime of roughly 0.2 seconds on a standard desktop. The transformation‑count SDF attains a slightly lower success rate (≈92 %) but still outperforms traditional length‑based attacks, which often fail to converge within feasible time limits. Even when the subgroup generators are deliberately chosen to increase combinatorial complexity, the distance‑guided search remains tractable, demonstrating robustness of the approach.

In the discussion, the paper emphasizes that the existence of efficiently computable SDFs poses a new class of threats to group‑based cryptography. Protocol designers must therefore consider subgroups whose elements are hard to approximate by any simple distance metric, or they must embed additional randomness (e.g., dynamic subgroup updates) to invalidate static distance evaluations. The authors also note that the SDF concept is not limited to Thompson’s group F; it can be adapted to other infinite non‑abelian groups such as braid groups, certain Artin groups, or higher‑dimensional Thompson‑like groups, potentially broadening the impact of their technique.

Finally, the paper concludes by summarizing the contributions: (1) the formal definition of subgroup distance functions, (2) two concrete, efficiently computable instantiations for Thompson’s group F, (3) a practical, low‑complexity attack algorithm against the Shpilrain‑Ushakov protocol, and (4) extensive experimental evidence of its superiority over prior methods. Future work is outlined, including the design of distance‑resistant subgroups, exploration of SDFs in other cryptographic primitives (signatures, zero‑knowledge proofs), and theoretical analysis of the hardness of approximating subgroup distance in various group families.