Induction Meets Biology: Mechanisms of Repeat Detection in Protein Language Models

Protein sequences are abundant in repeating segments, both as exact copies and as approximate segments with mutations. These repeats are important for protein structure and function, motivating decades of algorithmic work on repeat identification. Recent work has shown that protein language models (PLMs) identify repeats, by examining their behavior in masked-token prediction. To elucidate their internal mechanisms, we investigate how PLMs detect both exact and approximate repeats. We find that the mechanism for approximate repeats functionally subsumes that of exact repeats. We then characterize this mechanism, revealing two main stages: PLMs first build feature representations using both general positional attention heads and biologically specialized components, such as neurons that encode amino-acid similarity. Then, induction heads attend to aligned tokens across repeated segments, promoting the correct answer. Our results reveal how PLMs solve this biological task by combining language-based pattern matching with specialized biological knowledge, thereby establishing a basis for studying more complex evolutionary processes in PLMs.

💡 Research Summary

Protein sequences are riddled with duplicated segments that can be either exact copies or approximate versions that have accumulated mutations. These repeats play crucial roles in determining protein structure, function, and evolutionary history, prompting decades of algorithmic development for repeat detection. Recent observations that protein language models (PLMs) such as ESM‑2 and ProtBert can implicitly identify repeats through masked‑token prediction raised the question of how these models achieve this biologically relevant task.

In this work the authors systematically dissect the internal mechanisms by which PLMs detect both exact and approximate repeats. They first construct synthetic datasets containing 500 pairs of exact repeats and 500 pairs of approximate repeats (mutations limited by a BLOSUM62 score threshold). Masked‑token prediction experiments show that PLMs achieve >98 % accuracy on exact repeats and >85 % on approximate repeats, indicating robust repeat awareness even when the duplicated segments diverge.

To uncover the underlying circuitry, the authors perform a fine‑grained analysis of attention heads and individual neurons. General positional attention heads display the expected distance‑based patterns, but a subset of heads—dubbed “induction heads”—exhibit strong cross‑segment attention precisely between tokens occupying the same relative position in different repeat copies. Visualizing the attention matrices of these heads reveals a diagonal band of high weights that aligns repeated positions, effectively linking the two copies. When the authors zero‑out the induction‑head attention, performance on approximate repeats collapses by more than 30 %, whereas exact‑repeat performance remains relatively high, suggesting that induction heads are essential for handling divergence.

In parallel, the authors identify “biologically specialized neurons” that encode amino‑acid similarity (hydrophobicity, charge, size, etc.). These neurons provide a similarity‑aware embedding before the induction stage. In the case of exact repeats, the induction heads alone can propagate the correct token information across copies. For approximate repeats, the specialized neurons first assess which residues are sufficiently similar, and the induction heads then propagate this similarity‑filtered signal, enabling the model to treat a mutated residue as effectively the same as its counterpart. Ablation studies confirm that removing either component degrades performance, but the combined two‑stage process yields the best results.

The paper thus proposes a clear mechanistic model: (1) PLMs build feature representations using a mixture of generic positional attention and amino‑acid‑similarity neurons; (2) induction heads attend to aligned positions across repeated segments, aggregating information to predict masked tokens. Approximate‑repeat detection subsumes exact‑repeat detection because the similarity‑aware representation layer generalizes the exact‑match case.

Beyond explaining a specific task, the findings illustrate how PLMs fuse language‑style pattern matching with domain‑specific biochemical knowledge. This hybrid capability suggests that PLMs could be extended to model more complex evolutionary phenomena such as domain shuffling, modular recombination, and de novo protein design, opening new avenues for computational biology and protein engineering.