SOAP: Enhancing Spatio-Temporal Relation and Motion Information Capturing for Few-Shot Action Recognition

High frame-rate (HFR) videos of action recognition improve fine-grained expression while reducing the spatio-temporal relation and motion information density. Thus, large amounts of video samples are continuously required for traditional data-driven training. However, samples are not always sufficient in real-world scenarios, promoting few-shot action recognition (FSAR) research. We observe that most recent FSAR works build spatio-temporal relation of video samples via temporal alignment after spatial feature extraction, cutting apart spatial and temporal features within samples. They also capture motion information via narrow perspectives between adjacent frames without considering density, leading to insufficient motion information capturing. Therefore, we propose a novel plug-and-play architecture for FSAR called Spatio-tempOral frAme tuPle enhancer (SOAP) in this paper. The model we designed with such architecture refers to SOAP-Net. Temporal connections between different feature channels and spatio-temporal relation of features are considered instead of simple feature extraction. Comprehensive motion information is also captured, using frame tuples with multiple frames containing more motion information than adjacent frames. Combining frame tuples of diverse frame counts further provides a broader perspective. SOAP-Net achieves new state-of-the-art performance across well-known benchmarks such as SthSthV2, Kinetics, UCF101, and HMDB51. Extensive empirical evaluations underscore the competitiveness, pluggability, generalization, and robustness of SOAP. The code is released at https://github.com/wenbohuang1002/SOAP.

💡 Research Summary

The paper addresses two fundamental challenges that arise when applying few‑shot action recognition (FSAR) to high‑frame‑rate (HFR) videos. First, HFR footage, while providing finer‑grained visual cues, suffers from weakened spatio‑temporal relational cues and reduced motion‑information density, which in turn forces conventional data‑driven models to require far more training samples. Second, existing FSAR approaches typically extract spatial features first and then perform temporal alignment, effectively decoupling spatial and temporal information, and they capture motion only between adjacent frames, ignoring the richer motion cues that can be obtained from longer temporal windows.

To tackle these issues, the authors propose SOAP (Spatio‑tempOral frAme tuPle enhancer), a plug‑and‑play architecture that can be inserted into any FSAR pipeline. SOAP‑Net, the concrete model built with this architecture, consists of three parallel modules that each inject a specific prior into the raw video tensor before the backbone extracts features.

1. 3‑Dimension Enhancement Module (3DEM) – This module averages the input across channels to obtain a compact spatio‑temporal tensor, then applies a 3‑D convolution to jointly model spatial and temporal dependencies. The output is passed through a sigmoid gate and added residually to the original tensor, thereby providing a learned spatio‑temporal prior without destroying the original signal.

2. Channel‑Wise Enhancement Module (CWEM) – Inspired by Squeeze‑and‑Excitation, CWEM first performs spatial average pooling, expands the channel dimension with a 2‑D convolution, and then uses a 1‑D convolution across the temporal axis to adaptively recalibrate channel‑wise responses. After another 2‑D convolution restores the original channel count, a sigmoid‑gated residual connection yields a channel‑wise temporal prior.

3. Hybrid Motion Enhancement Module (HMEM) – Recognizing that motion density matters, HMEM moves beyond adjacent‑frame differencing. It defines a set O of tuple lengths (e.g., 2, 4, 6 frames) and, using a sliding‑window algorithm, extracts overlapping frame tuples of each length from both support and query videos. Each tuple branch processes its subsequence independently, and the resulting features are fused, delivering motion cues that capture both subtle displacements and larger movements.

These three priors are combined in parallel and fed into a standard 3‑D backbone (e.g., C3D, R(2+1)D). After backbone extraction, linear layers construct class prototypes from the support set, and a metric‑based distance (typically Euclidean) between query embeddings and prototypes yields the final classification. Because SOAP is modular, it can be attached to any existing FSAR method without retraining the backbone from scratch.

The authors evaluate SOAP‑Net on four widely used FSAR benchmarks: SthSthV2, Kinetics, UCF101, and HMDB51, under both 5‑way‑1‑shot and 5‑way‑5‑shot settings. SOAP‑Net consistently outperforms state‑of‑the‑art methods such as OTAM, TRX, HyRSM, and SloshNet, often by a large margin (e.g., +3–5% absolute accuracy). Ablation studies demonstrate that each module contributes positively: removing 3DEM degrades spatio‑temporal reasoning; omitting CWEM harms channel‑wise temporal consistency; discarding HMEM leads to the most significant drop, confirming the importance of multi‑scale motion tuples. Moreover, the model adds only modest computational overhead and remains compatible with various backbone architectures, confirming its claimed “pluggability” and “generalization”.

In summary, SOAP‑Net introduces a principled way to (i) jointly model spatial and temporal relations via 3‑D convolutions, (ii) adaptively calibrate channel‑wise temporal connections, and (iii) capture comprehensive motion information through multi‑scale frame tuples. This combination effectively mitigates the weaknesses of HFR video data in few‑shot scenarios, delivering new state‑of‑the‑art performance while remaining easy to integrate into existing FSAR systems.