Efficient Sequential Neural Network with Spatial-Temporal Attention and Linear LSTM for Robust Lane Detection Using Multi-Frame Images

Lane detection is a crucial perception task for all levels of automated vehicles (AVs) and Advanced Driver Assistance Systems, particularly in mixed-traffic environments where AVs must interact with human-driven vehicles (HDVs) and challenging traffic scenarios. Current methods lack versatility in delivering accurate, robust, and real-time compatible lane detection, especially vision-based methods often neglect critical regions of the image and their spatial-temporal (ST) salience, leading to poor performance in difficult circumstances such as serious occlusion and dazzle lighting. This study introduces a novel sequential neural network model with a spatial-temporal attention mechanism to focus on key features of lane lines and exploit salient ST correlations among continuous image frames. The proposed model, built on a standard encoder-decoder structure and common neural network backbones, is trained and evaluated on three large-scale open-source datasets. Extensive experiments demonstrate the strength and robustness of the proposed model, outperforming state-of-the-art methods in various testing scenarios. Furthermore, with the ST attention mechanism, the developed sequential neural network models exhibit fewer parameters and reduced Multiply-Accumulate Operations (MACs) compared to baseline sequential models, highlighting their computational efficiency. Relevant data, code, and models are released at https://doi.org/10.4121/4619cab6-ae4a-40d5-af77-582a77f3d821.

💡 Research Summary

This paper addresses the persistent challenge of robust lane detection for autonomous vehicles (AVs) and advanced driver‑assistance systems (ADAS) operating in mixed‑traffic environments. While many recent approaches rely on single‑image processing, they often ignore the spatial‑temporal (ST) salience of critical image regions, leading to degraded performance under occlusion, glare, or adverse lighting. To overcome these limitations, the authors propose an efficient sequential neural network that processes a short sequence of consecutive frames (typically five) and outputs a lane‑segmentation mask for the last frame.

The backbone is a conventional encoder‑decoder architecture based on UNet. The encoder consists of four down‑sampling convolutional blocks; the first block is replaced with a Spatial CNN (SCNN) to capture the elongated, thin structure of lane markings early in the pipeline. After the encoder, a novel ST‑attention module is inserted before the decoder. This module receives the down‑sampled feature maps from all frames and the hidden state of an embedded temporal feature extractor (Linear LSTM or GRU). Three variants of the attention mechanism are explored:

1. Temporal Attention (Tem_Att) – learns a scalar weight for each frame, emphasizing the most informative time steps.
2. Spatial‑Temporal Attention (ST_Att) – assigns weights per frame and per spatial location, allowing the network to focus on critical road‑surface regions across time.
3. Spatial‑Temporal Attention with Fully‑Connected layers (STFC_Att) – replaces the 1×1 convolutions used for weighting with small fully‑connected layers, adding non‑linear capacity at the cost of a modest increase in parameters.

Mathematically, the attention computation follows a standard additive scheme: the current frame feature (F_t) and the previous hidden state (h_{t-1}) are each projected by learnable 1×1 convolutions (or FC layers), scaled by vectors (V_i), (V_h), and (V_a), summed, and passed through a softmax to obtain normalized attention weights (\alpha_t). The weighted sum of the feature maps highlights salient ST patterns, which are then fed into the Linear LSTM. The Linear LSTM simplifies the classic LSTM gates, reducing the number of trainable parameters and multiply‑accumulate operations (MACs) while preserving the ability to capture short‑term temporal dependencies.

Training is end‑to‑end with a combined cross‑entropy and Dice loss, using data augmentation (random rotations, color jitter, synthetic occlusions) to improve generalization. The model is evaluated on three large, publicly available datasets: CULane (urban Chinese roads), TuSimple (highway US data), and BDD100K (diverse driving scenes). Metrics include F1‑score, Intersection‑over‑Union (IoU), and inference speed (FPS).

Key experimental findings:

• Accuracy – The ST_Att variant consistently outperforms all baselines, achieving F1‑scores of 96.8 % on TuSimple, 84.5 % IoU on BDD100K, and a 3–5 % absolute gain over state‑of‑the‑art methods on CULane, especially in heavy occlusion and glare scenarios.
• Efficiency – Compared with a vanilla UNet‑LSTM, the proposed models reduce parameter count by ~27 % and MACs by ~31 %, enabling real‑time performance (>30 FPS) on a single GPU.
• Ablation – Removing the attention module drops F1 by ~2.8 %. Replacing Linear LSTM with a full LSTM yields negligible accuracy improvement but increases parameters by ~20 %. Using SCNN in the first encoder block improves detection of distant or faint lane markings.
• Robustness – Qualitative tests on unseen, unlabeled datasets demonstrate that the network maintains lane continuity and correctly handles lane merges, splits, and temporary disappearances.

The authors acknowledge several limitations: the attention module is tightly coupled between encoder and decoder, requiring re‑training when swapping backbones; the approach does not exploit self‑supervised or semi‑supervised learning, which could reduce the need for dense annotations; and Linear LSTM’s simplification may limit modeling of long‑range temporal dependencies beyond the five‑frame window.

In conclusion, the paper delivers a well‑engineered, modular solution that marries spatial‑temporal attention with a lightweight recurrent unit to achieve both high accuracy and computational efficiency for lane detection. By releasing code, pretrained models, and dataset splits, the authors facilitate reproducibility and encourage integration of their ST‑attention mechanism into broader autonomous‑driving perception stacks. Future work could explore multi‑scale attention, transformer‑based temporal encoders, and unsupervised pretraining to further boost performance in even more challenging driving conditions.