Bidirectional Representations Augmented Autoregressive Biological Sequence Generation

Autoregressive (AR) models, common in sequence generation, are limited in many biological tasks such as de novo peptide sequencing and protein modeling by their unidirectional nature, failing to capture crucial global bidirectional token dependencies. Non-Autoregressive (NAR) models offer holistic, bidirectional representations but face challenges with generative coherence and scalability. To transcend this, we propose a hybrid framework enhancing AR generation by dynamically integrating rich contextual information from non-autoregressive mechanisms. Our approach couples a shared input encoder with two decoders: a non-autoregressive one learning latent bidirectional biological features, and an AR decoder synthesizing the biological sequence by leveraging these bidirectional features. A novel cross-decoder attention module enables the AR decoder to iteratively query and integrate these bidirectional features, enriching its predictions. This synergy is cultivated via a tailored training strategy with importance annealing for balanced objectives and cross-decoder gradient blocking for stable, focused learning. Evaluations on a demanding nine-species benchmark of de novo peptide sequencing show that our model substantially surpasses AR and NAR baselines. It uniquely harmonizes AR stability with NAR contextual awareness, delivering robust, superior performance on diverse downstream data. This research advances biological sequence modeling techniques and contributes a novel architectural paradigm for augmenting AR models with enhanced bidirectional understanding for complex sequence generation. Code is available at https://github.com/BEAM-Labs/denovo.

💡 Research Summary

The paper introduces a groundbreaking hybrid framework designed to overcome the inherent limitations of existing architectures in biological sequence generation, specifically targeting tasks like de novo peptide sequencing and protein modeling. Traditionally, Autoregressive (AR) models have been the standard for sequence generation due to their ability to produce coherent and stable sequences. However, their unidirectional nature prevents them from capturing the global, bidirectional dependencies that are critical to the structural integrity of biological molecules. On the other mutally exclusive side, Non-Autoregressive (NAR) models provide holistic, bidirectional representations that capture global context, yet they struggle with maintaining generative coherence and scaling to complex sequences.

To bridge this gap, the authors propose a novel architecture that augments the generative stability of AR models with the contextual richness of NAR models. The proposed framework utilizes a shared input encoder coupled with two distinct decoders. The first is a Non-Autoregressive decoder, which focuses on learning and extracting latent bidirectional biological features from the input. The second is an Autoregressive decoder, which is responsible for the actual synthesis of the biological sequence. The core innovation lies in the “Cross-decoder Attention Module,” which allows the AR decoder to dynamically query and integrate the bidirectional features extracted by the NAR decoder. This mechanism ensures that the generative process is informed by the global structural context of the entire sequence.

A significant challenge in training such a dual-decoder system is the potential for optimization imbalance and instability. To address this, the researchers implemented a sophisticated training strategy involving two key techniques: “Importance Annealing” and “Cross-decoder Gradient Blocking.” Importance annealing is used to balance the competing objectives of the two decoders, ensuring that neither the feature extraction nor the sequence generation task dominates the learning process prematurely. Cross-decoder gradient blocking is employed to prevent the AR decoder from simply mimicking the NAR decoder’s outputs, thereby forcing it to learn how to effectively utilize the provided features to generate accurate, novel sequences.

The effectiveness of this approach was rigorously evaluated using a demanding nine-species benchmark for de novo peptide sequencing. The experimental results demonstrate that the proposed model substantially outperforms both standalone AR and NAR baselines. By successfully harmonizing the stability of AR generation with the contextual awareness of NAR representations, the model provides a robust and superior solution for complex biological sequence modeling. This research establishes a new architectural paradigm, offering significant implications for the fields of protein engineering, drug discovery, and synthetic biology, where understanding and generating complex, interdependent sequences is paramount.