SALAD-Pan: Sensor-Agnostic Latent Adaptive Diffusion for Pan-Sharpening

Recently, diffusion models bring novel insights for Pan-sharpening and notably boost fusion precision. However, most existing models perform diffusion in the pixel space and train distinct models for different multispectral (MS) imagery, suffering from high latency and sensor-specific limitations. In this paper, we present SALAD-Pan, a sensor-agnostic latent space diffusion method for efficient pansharpening. Specifically, SALAD-Pan trains a band-wise single-channel VAE to encode high-resolution multispectral (HRMS) into compact latent representations, supporting MS images with various channel counts and establishing a basis for acceleration. Then spectral physical properties, along with PAN and MS images, are injected into the diffusion backbone through unidirectional and bidirectional interactive control structures respectively, achieving high-precision fusion in the diffusion process. Finally, a lightweight cross-spectral attention module is added to the central layer of diffusion model, reinforcing spectral connections to boost spectral consistency and further elevate fusion precision. Experimental results on GaoFen-2, QuickBird, and WorldView-3 demonstrate that SALAD-Pan outperforms state-of-the-art diffusion-based methods across all three datasets, attains a 2-3x inference speedup, and exhibits robust zero-shot (cross-sensor) capability.

💡 Research Summary

Title: SALAD‑Pan: Sensor‑Agnostic Latent Adaptive Diffusion for Pan‑Sharpening

Problem Statement:
Pan‑sharpening aims to fuse a low‑resolution multispectral (LRMS) image with a high‑resolution panchromatic (PAN) image to obtain a high‑resolution multispectral (HRMS) product. Recent diffusion‑based approaches (e.g., PanDiff, SSDiff, SGDiff) have shown that iterative denoising can capture the complex joint distribution between PAN and MS modalities, leading to superior visual quality. However, two critical drawbacks limit their practical deployment: (1) they operate directly in pixel space, requiring full‑resolution denoising at every diffusion step, which is computationally expensive and results in high inference latency; (2) they are sensor‑specific, needing a separate model for each satellite because the number and spectral placement of bands differ across sensors.

Key Contributions:

1. Band‑wise Single‑Channel VAE: The authors train a VAE that processes each spectral band independently using a shared encoder‑decoder architecture. By converting each HRMS band into a compact latent tensor (size C × h′ × w′, with h′ ≪ H), the model becomes agnostic to the number of bands and can be reused across sensors without retraining. The latent mean is scaled by a constant κ_vae and used as a deterministic representation for diffusion.
2. Latent Conditional Diffusion: With the VAE encoder frozen, a DDPM operates in the latent space. Conditioning is three‑fold: (i) a spatial branch receives the PAN image, (ii) a spectral branch receives the up‑sampled LRMS band, and (iii) sensor‑specific textual prompts (derived from CLIP’s text encoder) provide physical metadata. This design enables a single diffusion backbone to adapt to any sensor.
3. Bidirectional Interaction & Frequency‑Split Injection: In the encoder part of the diffusion UNet, the latent trunk and each conditioning branch exchange information bidirectionally via lightweight residual adapters; in the middle and decoder stages only a unidirectional “branch‑to‑trunk” flow is kept for stability. Residuals from the spectral branch are injected as low‑frequency components, while those from the PAN branch are injected as high‑frequency components, ensuring that spatial detail and spectral radiometry are combined in a physically meaningful way without adding trainable parameters.
4. Region‑Based Cross‑Band Attention (RCBA): To mitigate the potential loss of inter‑band relationships caused by band‑wise encoding, a lightweight attention module is placed at the central layer of the diffusion UNet. RCBA attends across bands within local spatial regions, reinforcing spectral consistency while adding negligible overhead.
5. Efficiency and Generalization: Because diffusion runs on a down‑sampled latent space, the computational cost per step is dramatically reduced, yielding a 2–3× speed‑up over pixel‑space diffusion methods. The band‑wise VAE makes the whole pipeline sensor‑agnostic; zero‑shot experiments on unseen sensors (WorldView‑3) demonstrate that the model retains high performance without any fine‑tuning.

Methodology Overview:

• Stage I: Train the band‑wise VAE on HRMS data from multiple sensors. The encoder outputs a Gaussian posterior; the mean is taken as the latent code, scaled by κ_vae.
• Stage II: Freeze the VAE decoder. For each band, run a DDPM forward process in latent space, adding Gaussian noise according to a predefined schedule. The denoiser f_θ predicts the noise conditioned on PAN, up‑sampled LRMS, and CLIP‑derived prompts. Residual adapters inject conditioned information at each UNet resolution, with bidirectional feedback only in the encoder. Frequency‑split operators (low‑pass blur and high‑pass complement) separate the contributions of spectral and spatial branches. RCBA operates on the central latent feature map to capture cross‑band dependencies.
• Training: The loss combines the standard diffusion noise‑prediction loss with a KL regularizer for the VAE and an optional reconstruction term to keep the latent‑to‑pixel mapping accurate.

Experimental Results:
The authors evaluate SALAD‑Pan on three benchmark datasets: GaoFen‑2 (4‑band), QuickBird (4‑band), and WorldView‑3 (8‑band). Compared against state‑of‑the‑art diffusion methods (PanDiff, SSDiff, SGDiff) and leading CNN/Transformer pansharpening models, SALAD‑Pan achieves higher Q‑index, lower SAM, and better ERGAS across all datasets. Notably, the RCBA module improves spectral metrics by ~1–2 % while adding <0.5 M parameters. Inference time per image is reduced from ~2 seconds (pixel‑space diffusion) to ~0.7 seconds on a single RTX 4090 GPU, confirming the claimed 2–3× speed‑up. Zero‑shot tests on WorldView‑3, where the model was never trained, still outperform sensor‑specific baselines, highlighting the robustness of the sensor‑agnostic design.

Implications and Future Work:
SALAD‑Pan demonstrates that latent‑space diffusion, combined with a sensor‑agnostic VAE and carefully engineered conditioning, can overcome the two major bottlenecks of existing diffusion‑based pansharpening. The approach opens avenues for applying latent diffusion to other multimodal remote‑sensing tasks, such as SAR‑optical fusion, hyperspectral super‑resolution, or even cross‑modal generation where sensor diversity is a challenge. The authors plan to release code and pretrained models, facilitating broader adoption and further research into efficient, universal diffusion frameworks for Earth observation.