CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis

Spectral imaging offers promising applications across diverse domains, including medicine and urban scene understanding, and is already established as a critical modality in remote sensing. However, variability in channel dimensionality and captured wavelengths among spectral cameras impede the development of AI-driven methodologies, leading to camera-specific models with limited generalizability and inadequate cross-camera applicability. To address this bottleneck, we introduce CARL, a model for Camera-Agnostic Representation Learning across RGB, multispectral, and hyperspectral imaging modalities. To enable the conversion of a spectral image with any channel dimensionality to a camera-agnostic representation, we introduce a novel spectral encoder, featuring a self-attention-cross-attention mechanism, to distill salient spectral information into learned spectral representations. Spatio-spectral pre-training is achieved with a novel feature-based self-supervision strategy tailored to CARL. Large-scale experiments across the domains of medical imaging, autonomous driving, and satellite imaging demonstrate our model’s unique robustness to spectral heterogeneity, outperforming on datasets with simulated and real-world cross-camera spectral variations. The scalability and versatility of the proposed approach position our model as a backbone for future spectral foundation models. Code and model weights are publicly available at https://github.com/IMSY-DKFZ/CARL.

💡 Research Summary

The paper introduces CARL (Camera‑Agnostic Representation Learning), a unified framework for processing spectral images regardless of the camera’s channel count or wavelength coverage. Existing deep‑learning models for spectral data (e.g., CNNs, Vision Transformers) assume a fixed number of channels and therefore must be retrained for each sensor type, limiting cross‑camera generalization. CARL tackles this problem by separating spectral and spatial processing and by designing a self‑attention‑cross‑attention encoder that can ingest any number of spectral bands and produce a compact, camera‑agnostic representation.

Spectral Encoder (E_spec).
Given an input image I ∈ ℝ^{H×W×C} with arbitrary C, the image is divided into P×P patches. Each patch is projected channel‑wise by a shared 2‑D convolution into a D‑dimensional token Λ_i (i = 1…C). To align channels from different cameras, the wavelength λ_i of each band is encoded with sinusoidal Fourier features, producing a positional encoding PE(λ_i) ∈ ℝ^{D}. Adding PE(λ_i) to Λ_i injects wavelength information directly into the token space, enabling the model to learn correspondences across sensors with different spectral ranges.

The encoder then applies a standard transformer self‑attention block over the set of spectral tokens {Λ_i}. This captures inter‑band relationships. Afterwards, K learnable spectral “prototype” vectors S_j (j = 1…K) attend to the tokens via a cross‑attention block. The cross‑attention aggregates the most informative spectral cues into the K prototypes, effectively compressing an arbitrary‑size spectral dimension into a fixed‑size representation. The process (self‑attention → cross‑attention) is repeated L times, allowing iterative refinement. Finally, the K prototypes are summed (or otherwise pooled) to obtain a camera‑agnostic feature for the patch, which is then fed to a conventional spatial encoder (e.g., ViT). Because the spectral encoder has already removed device‑specific wavelength information, any off‑the‑shelf spatial transformer can be used without modification.

Self‑Supervised Pre‑training (CARL‑SSL).
CARL adopts a teacher‑student paradigm with exponential moving average (EMA) updates for the teacher. The spatial component uses I‑JEPA, a state‑of‑the‑art masked‑prediction SSL method for vision transformers. For the spectral part, the authors devise a novel feature‑based SSL: a random subset M of spectral bands is masked in the student input, while the teacher receives the full set. The student processes the unmasked tokens, producing the K prototypes; the teacher produces the full set of spectral tokens and prototypes. A transformer predictor ϕ_spec receives the student prototypes together with learned mask tokens (augmented with the positional encodings of the masked wavelengths) and is trained to reconstruct the teacher’s masked spectral tokens. This encourages the student to infer missing spectral information from the learned prototypes, thereby learning robust, wavelength‑aware embeddings without any pixel‑level supervision.

Key Contributions.

1. First method to achieve truly camera‑agnostic spatio‑spectral encoding by combining wavelength positional encoding, self‑attention, and cross‑attention.
2. A novel spectral feature‑based SSL that can be combined with I‑JEPA for end‑to‑end spatio‑spectral pre‑training on large unlabeled datasets.
3. Extensive cross‑domain validation on medical endoscopy (HSI+MSI), autonomous‑driving street scenes (RGB+MSI), and satellite remote sensing (HSI). CARL consistently outperforms camera‑specific baselines, channel‑invariant baselines, and recent spectral‑only models, achieving 4–8 % absolute gains in mIoU/accuracy on cross‑camera test splits.

Experimental Findings.
The authors evaluate five public datasets spanning three domains, each containing images from multiple sensors with differing band counts (e.g., 3‑band RGB, 8‑band MSI, 200‑band HSI). They train CARL on a mixture of these datasets and test on unseen sensors. Results show that CARL’s performance degrades only minimally when the test sensor’s wavelength range differs dramatically from the training sensors, confirming the effectiveness of the wavelength positional encoding and the prototype‑based compression. Ablation studies demonstrate that both self‑attention and cross‑attention are necessary; removing either reduces performance by 2–3 %. The SSL component also yields a 3 % boost over purely supervised fine‑tuning, highlighting the benefit of large‑scale unlabeled pre‑training.

Limitations and Future Work.
CARL introduces hyper‑parameters K (number of spectral prototypes) and L (number of attention iterations) that may need tuning for new domains. The current design processes each patch independently, which may limit the capture of global spectral‑spatial dependencies. Future research directions include multi‑scale cross‑attention, dynamic prototype allocation, and model compression for real‑time deployment.

Overall Impact.
CARL provides the first general‑purpose backbone that can ingest any spectral image—whether RGB, multispectral, or hyperspectral—and output a unified representation suitable for downstream tasks such as segmentation, classification, or detection. By decoupling spectral variability from spatial modeling and by leveraging a tailored self‑supervised pre‑training scheme, CARL opens the door to “spectral foundation models” that can be trained on massive, heterogeneous sensor collections and then fine‑tuned for specific applications without the need for sensor‑specific retraining. This represents a significant step toward unifying the fragmented landscape of spectral imaging AI.