Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition with Multimodal Training

We present an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition. Instead of explicitly combining multimodal information, which is commonplace in many state-of-the-art methods, we propose a different framework in which we embed the knowledge of multiple modalities in individual networks so that each unimodal network can achieve an improved performance. In particular, we dedicate separate networks per available modality and enforce them to collaborate and learn to develop networks with common semantics and better representations. We introduce a “spatiotemporal semantic alignment” loss (SSA) to align the content of the features from different networks. In addition, we regularize this loss with our proposed “focal regularization parameter” to avoid negative knowledge transfer. Experimental results show that our framework improves the test time recognition accuracy of unimodal networks, and provides the state-of-the-art performance on various dynamic hand gesture recognition datasets.

💡 Research Summary

The paper introduces a novel training paradigm for dynamic hand‑gesture recognition that leverages multiple sensor modalities (RGB, depth, optical flow) while retaining the ability to operate with a single modality at test time. Instead of the conventional multimodal fusion where all streams are concatenated or jointly processed during both training and inference, the authors propose a “Multimodal‑Training/Unimodal‑Testing” (MTUT) framework. For each available modality a separate 3‑D convolutional neural network (3D‑CNN) is instantiated. During training each network receives only its own modality data and is supervised by the standard classification loss. In addition, a new “Spatiotemporal Semantic Alignment” (SSA) loss is introduced to align the internal representations of the different networks.

The SSA loss operates on a deep feature map (F_m\in\mathbb{R}^{W\times H\times T\times C}) from an intermediate layer of network (m). The feature map is reshaped into a matrix of size (d\times C) (where (d=WHT) is the number of spatiotemporal positions) and rows are (\ell_2)‑normalized. The correlation matrix (\text{corr}(F_m)=\hat{F}_m\hat{F}_m^{\top}) captures the pairwise relationships between all spatiotemporal positions while preserving their ordering. For any pair of modalities ((m,n)) the SSA loss is defined as the squared Frobenius norm (| \text{corr}(F_m)-\text{corr}(F_n) |_F^2). By minimizing this term the networks are encouraged to develop a common semantic understanding of the same video content, even though they process different visual cues.

A key challenge is that not all modalities are equally informative for every frame. For static scenes RGB may be superior, while for fast motion optical flow may be more reliable. Blindly aligning the representations could therefore cause “negative transfer,” where a weaker modality is forced to imitate a stronger one and performance degrades. To prevent this, the authors propose a “focal regularization parameter” (\rho_{m,n}) that weights the SSA loss based on the relative classification losses of the two networks. Let (\ell^{\text{cls}}_m) and (\ell^{\text{cls}}_n) be the current classification losses; define (\Delta\ell = \ell^{\text{cls}}m - \ell^{\text{cls}}n). If (\Delta\ell>0) (network (n) is currently better), (\rho{m,n}) is set to a positive value that grows with (\Delta\ell); otherwise (\rho{m,n}=0). Concretely, \