MCGA: Mixture of Codebooks Hyperspectral Reconstruction via Grayscale-Aware Attention

Reconstructing hyperspectral images (HSIs) from RGB inputs provides a cost-effective alternative to hyperspectral cameras, but reconstructing high-dimensional spectra from three channels is inherently ill-posed. Existing methods typically directly regress RGB-to-HSI mappings using large attention networks, which are computationally expensive and handle ill-posedness only implicitly. We propose MCGA, a Mixture-of-Codebooks with Grayscale-aware Attention framework that explicitly addresses these challenges using spectral priors and photometric consistency. MCGA first learns transferable spectral priors via a mixture-of-codebooks (MoC) from heterogeneous HSI datasets, then aligns RGB features with these priors through grayscale-aware photometric attention (GANet). Efficiency and robustness are further improved via top-K attention design and test-time adaptation (TTA). Experiments on multiple real-world benchmarks demonstrate the state-of-the-art accuracy, strong cross-dataset generalization, and 4-5x faster inference. Codes will be available once acceptance at https://github.com/Fibonaccirabbit/MCGA.

💡 Research Summary

The paper tackles the challenging problem of reconstructing hyperspectral images (HSIs) from ordinary RGB photographs, a task that is fundamentally ill‑posed because three broadband channels must be mapped to dozens or hundreds of narrow spectral bands. Existing approaches either rely on large attention‑heavy networks that are computationally demanding or on residual/dense CNNs that struggle with illumination, sensor, and noise variations. To overcome these limitations, the authors propose MCGA (Mixture‑of‑Codebooks with Grayscale‑aware Attention), a two‑stage framework that explicitly injects physical priors and photometric consistency into the reconstruction pipeline.

Stage 1 – Spectral Prior Learning. A multi‑scale vector‑quantized variational auto‑encoder (VQ‑VAE) is trained separately on several heterogeneous HSI datasets (HySpecNet‑11k, ARAD‑1k, HyperGlobal‑450K). For each dataset a set of codebooks is learned; each codebook contains 512‑dimensional embedding vectors for roughly half of the spectral channels. The codebooks are concatenated to form a “Mixture of Codebooks” (MoC) that captures cross‑dataset spectral diversity while remaining transferable to unseen data. The VQ‑VAE loss combines a reconstruction term with the standard embedding and commitment losses.

Stage 2 – Grayscale‑aware Reconstruction Network (GANet). GANet receives an RGB image, extracts features with a transformer‑style encoder‑decoder, and aligns them to the MoC. Two novel grayscale‑aware operations are introduced: a learnable γ‑correction (GAγ) that raises the input to a power a, and a learnable logarithmic transform (GAl) that scales the log of the input. The parameters a are derived from a softmax‑MLP applied to the global average of the normalized feature map, allowing the network to adaptively control brightness and grayscale scaling.

Top‑K Quantized Attention. Traditional self‑attention scales as O(C²HW), which is prohibitive for high‑dimensional HSIs. MCGA reduces this cost by selecting the top‑K most frequently used quantized vectors from the codebooks (based on hit rates) and using only these vectors to compute queries and keys. Consequently, the complexity drops to O(C²K) with K ≪ HW, delivering a 4–5× speedup while preserving accuracy.

Test‑Time Adaptation (TTA). Real‑world deployments encounter illumination shifts, sensor response changes, and scene distribution drifts. To adapt without ground‑truth HSIs, the authors minimize the entropy of the MoC assignment probabilities (−∑ P log P) during inference, updating only the affine parameters of the grayscale‑aware modules while freezing the encoder, decoder, and codebooks. This lightweight adaptation aligns RGB features to the MoC manifold under new conditions.

Experimental Validation. The method is evaluated on two large‑scale benchmarks: ARAD‑1k (31 bands, 400–700 nm) and HySpecNet‑11k (224 bands, 420–2450 nm). MCGA‑S2 (the two‑scale variant) achieves state‑of‑the‑art performance: on ARAD‑1k it reduces RMSE from 0.0248 (MST++) to 0.0182 (≈27 % improvement) and MRAE from 0.0248 to 0.0131, while inference time drops from ~435 ms to ~94 ms (≈4.6× faster). On HySpecNet‑11k it outperforms the previous best R3ST by 13 % in RMSE and 1.6 % in MRAE, with a 5× speedup.

Robustness Tests. In a “mixed” setting where spatial layouts are shuffled (breaking spatial correlations), most attention‑based baselines suffer severe degradation, whereas MCGA’s performance degrades only modestly, demonstrating that the pixel‑level MoC encoding and grayscale‑aware attention are largely spatial‑agnostic. Illumination perturbation experiments (γ = 0.9 and 1.1) show that MCGA‑S2+TTA reduces MRAE by roughly 10 % compared to the non‑adapted version, confirming the effectiveness of entropy‑based TTA.

Ablation Studies. Removing the mixture of codebooks (using a single codebook) raises MRAE by ~16 %; omitting grayscale‑aware modules adds another ~10 % error; replacing top‑K attention with full attention yields negligible accuracy change but dramatically increases computation.

Contributions and Impact. MCGA introduces (1) a transferable spectral prior via a multi‑scale VQ‑VAE mixture of codebooks, (2) a grayscale‑aware attention mechanism that explicitly models intensity variations, (3) an efficient top‑K quantized attention scheme, and (4) a lightweight test‑time adaptation strategy. Together these components deliver a solution that is simultaneously more accurate, faster, and more robust than prior art. The authors also suggest that the MoC could be repurposed for synthetic HSI generation or data augmentation, and that grayscale‑aware attention may benefit other low‑quality image restoration tasks.

In summary, MCGA represents a significant step forward in RGB‑to‑HSI reconstruction, offering a practical, high‑performance alternative for real‑world applications where hyperspectral cameras are impractical.