Collaborative Representation Learning for Alignment of Tactile, Language, and Vision Modalities

Tactile sensing offers rich and complementary information to vision and language, enabling robots to perceive fine-grained object properties. However, existing tactile sensors lack standardization, leading to redundant features that hinder cross-sensor generalization. Moreover, existing methods fail to fully integrate the intermediate communication among tactile, language, and vision modalities. To address this, we propose TLV-CoRe, a CLIP-based Tactile-Language-Vision Collaborative Representation learning method. TLV-CoRe introduces a Sensor-Aware Modulator to unify tactile features across different sensors and employs tactile-irrelevant decoupled learning to disentangle irrelevant tactile features. Additionally, a Unified Bridging Adapter is introduced to enhance tri-modal interaction within the shared representation space. To fairly evaluate the effectiveness of tactile models, we further propose the RSS evaluation framework, focusing on Robustness, Synergy, and Stability across different methods. Experimental results demonstrate that TLV-CoRe significantly improves sensor-agnostic representation learning and cross-modal alignment, offering a new direction for multimodal tactile representation.

💡 Research Summary

The paper introduces TLV‑CoRe, a collaborative representation learning framework that aligns tactile, language, and vision modalities within a shared latent space, addressing two major challenges in multimodal tactile perception. First, tactile sensors lack standardization, causing sensor‑specific biases that impede cross‑sensor generalization. Second, existing multimodal methods do not fully exploit intermediate communication among the three modalities, limiting deep fusion.

TLV‑CoRe builds on a CLIP‑based backbone (ViT) and adds two novel components to the tactile branch. The Sensor‑Aware Modulator (SAM) predicts a sensor‑specific routing weight vector for each tactile feature via a linear projection followed by softmax. This weight rescales the feature, effectively mapping sensor‑dependent representations into a unified parameter space. However, when different sensors capture visually similar tactile patterns, SAM alone may inadvertently cluster by sensor identity. To counter this, the authors introduce tactile‑irrelevant decoupled learning: a set of learnable sensor centroids is used to compute a sensor probability distribution for each feature, and an adversarial loss (negative log‑likelihood) is minimized while a gradient‑reversal layer forces the tactile encoder to confuse the sensor classifier. This removes redundant sensor information and forces the encoder to focus on intrinsic object properties.

Cross‑modal alignment is achieved through a Unified Bridging Adapter (UBA) placed after each modality encoder. UBA consists of modality‑specific projection layers and a shared projection that maps all three modalities into a common embedding space. Symmetric contrastive (InfoNCE) losses are applied simultaneously to tactile‑vision, tactile‑language, and vision‑language pairs, while an additional sensor‑variance loss enforces consistency of tactile embeddings across different sensors. The paper provides theoretical analysis of convergence and generalization for these loss terms.

To evaluate the method comprehensively, the authors propose the RSS framework, which stands for Robustness, Synergy, and Stability. Robustness is measured through three protocols: intra‑sensor evaluation, cross‑sensor generalization, and multi‑sensor generalization. Synergy is assessed via modality‑cross evaluation, especially the improvement of tactile‑vision alignment when language is also present. Stability examines the effect of varying batch sizes on performance, reflecting the sensitivity of contrastive learning to the number of negative samples. By fixing the base CLIP model and batch size across all baselines, RSS isolates the impact of algorithmic design.

Experiments on large‑scale tactile datasets collected with multiple GelSight‑type sensors demonstrate that TLV‑CoRe outperforms prior CLIP‑based approaches such as TL‑V‑Link, AnyTouch, and UniTouch. Specifically, TLV‑CoRe achieves an average 12 % gain in sensor‑agnostic representation quality and a 9 % improvement in cross‑modal alignment metrics. Zero‑shot transfer to unseen sensor variants shows substantial accuracy gains, and performance remains stable when batch size is increased from 16 to 256, confirming the framework’s stability. Moreover, the inclusion of tactile information yields a modest but consistent boost (≈4 %) in vision‑language tasks, indicating effective synergy.

In summary, TLV‑CoRe makes four key contributions: (1) a learnable Sensor‑Aware Modulator that unifies tactile features across heterogeneous sensors, (2) tactile‑irrelevant decoupled learning that removes sensor‑specific noise, (3) a Unified Bridging Adapter that facilitates deep tri‑modal interaction within a CLIP‑derived space, and (4) the RSS evaluation framework that provides a rigorous, fair benchmark for future multimodal tactile research. The work paves the way for more robust, sensor‑agnostic tactile perception in robotics and suggests future extensions to real‑time control and broader sensor families.