Patient-Aware Multimodal RGB-HSI Fusion via Incremental Heuristic Meta-Learning for Oral Lesion Classification

Early detection of oral cancer and potentially malignant diseases is a major challenge in low-resource settings due to the scarcity of annotated data. We provide a unified approach for four-class oral lesion classification that incorporates deep learning, spectral analysis, and demographic data. A pathologist-verified subset of oral cavity images was curated from a publicly available dataset. Oral cavity pictures were processed using a fine-tuned ConvNeXt-v2 network for deep embeddings before being translated into the hyperspectral domain using a reconstruction algorithm. Haemoglobin-sensitive, textural, and spectral descriptors were obtained from the reconstructed hyperspectral cubes and combined with demographic data. Multiple machine-learning models were evaluated using patient-specific validation. Finally, an incremental heuristic meta-learner (IHML) was developed that merged calibrated base classifiers via probabilistic feature stacking and uncertainty-aware abstraction of multimodal representations with patient-level smoothing. By decoupling evidence extraction from decision fusion, IHML stabilizes predictions in heterogeneous, small-sample medical datasets. On an unseen test set, our proposed model achieved a macro F1 of 66.23% and an overall accuracy of 64.56%. The findings demonstrate that RGB-to-hyperspectral reconstruction and ensemble meta-learning improve diagnostic robustness in real-world oral lesion screening.

💡 Research Summary

The paper tackles the pressing problem of early oral cancer and potentially malignant disorder (OPMD) detection in low‑resource settings, where annotated data are scarce and specialized equipment is unavailable. The authors propose a unified, patient‑aware multimodal framework that transforms ordinary RGB photographs of the oral cavity into synthetic hyperspectral images (HSI), extracts deep and handcrafted features, and combines them with demographic information to classify lesions into four categories: healthy, benign, OPMD, and oral cancer (OCA).

Dataset
A publicly available oral lesion dataset containing 3,000 white‑light images from 714 patients is curated by a pathologist, discarding low‑quality or diagnostically ambiguous samples. After filtering, 2,438 images from 653 patients remain, with a naturally imbalanced class distribution (OPMD 46.5 %, benign 24.9 %, healthy 24.3 %, OCA 4.3 %). Each image is cropped to the lesion region, resized to 512 × 512 px, and accompanied by five demographic variables (age, gender, smoking, alcohol, betel‑chewing).

Hyperspectral Reconstruction
To avoid costly hyperspectral cameras, the authors fine‑tune the Multi‑Stage Progressive Image Restoration Network (MPRNet) to generate a 31‑band HSI cube (400–700 nm, 10 nm intervals) from each RGB ROI. The reconstruction achieves a peak signal‑to‑noise ratio (PSNR) of 33.5 dB, which the authors deem sufficient for downstream feature extraction.

Feature Extraction

1. Deep Morphological Embedding – A ConvNeXt‑v2 model pre‑trained on ImageNet is fine‑tuned on the curated set; the global average‑pooled output (768‑dimensional) serves as a high‑level visual descriptor.
2. Hemoglobin‑Sensitive Spectral Biomarkers – Ratios and normalized differences that capture the characteristic absorption of oxy‑ and deoxy‑hemoglobin (e.g., R545/R575, NDI) are computed across the synthetic HSI, yielding 46 features.
3. Multiscale Texture Descriptors – GLCM statistics, Local Binary Patterns, Gabor filter responses, and SIFT keypoint histograms provide 58 texture features.
4. Spectral‑Shape & Unmixing Features – Band‑wise maxima/minima, peak‑to‑valley amplitudes, slopes, and curvature measures (31 features) describe the overall reflectance curve, reflecting chromophore composition such as keratin and melanin.
5. Demographic Metadata – The five clinical risk factors are one‑hot or normalized and appended as a 5‑dimensional vector.

All modalities are independently normalized and concatenated into a single multimodal vector x = z_deep ∥ z_hae ∥ z_tex ∥ z_spec ∥ z_demo.

Incremental Heuristic Meta‑Learner (IHML)
The core of the system is a two‑stage meta‑learning architecture:

Base Learners – Four calibrated classifiers (LightGBM, Extra Trees, Gradient Boosting, isotonic‑calibrated Logistic Regression) are trained on x, each producing a probability distribution p(m) over the four classes.

Uncertainty‑Aware Meta‑Features – From each p(m) the authors derive scalar confidence statistics: highest class probability, margin between top two classes, and Shannon entropy. These form a low‑dimensional vector c(m).

Meta‑Vector Construction – All base probabilities and confidence vectors are stacked: h = Φ(p(1)…p(M), c(1)…c(M)).

Patient‑Level Posterior Smoothing – Because multiple images belong to the same patient, the method iteratively blends each sample’s probability with the mean probability of its patient group:
p_{t+1}(i) = (1 − α) p_t(i) + α p_g(i),
where α (≈ 0.3) controls the influence of the group prior. This Bayesian‑like smoothing enforces intra‑patient label consistency.

Meta‑Classifier – A multinomial logistic regression is trained on h to output the final class prediction.

By operating entirely in probability space, IHML decouples evidence extraction from decision fusion, explicitly models predictive uncertainty, and mitigates feature redundancy and class imbalance.

Experimental Protocol
A patient‑wise split is used: 15 % of patients form an unseen test set, while the remaining 85 % constitute the development set. StratifiedGroupKFold (5‑fold) ensures that images from the same patient never appear in both training and validation folds.

Results
The authors benchmark a wide range of baselines: traditional machine‑learning models (logistic regression, random forest, SVM, XGBoost, LightGBM) and recent deep‑tabular architectures (TabICL, T2G‑Former, TabTransformer, DANet). IHML consistently outperforms all competitors across macro F1, per‑class recall, and overall accuracy. On the held‑out patient test set, IHML achieves a macro F1 of 66.23 % and an overall accuracy of 64.56 %, surpassing the best deep‑tabular model by several percentage points. Notably, the smoothing step improves stability for the minority OCA class, reducing intra‑patient prediction variance.

Discussion and Limitations
The study’s primary contribution is a practical pipeline that leverages inexpensive RGB imaging to obtain spectrally enriched cues, combines them with powerful deep embeddings and clinical priors, and fuses heterogeneous classifiers through uncertainty‑aware meta‑learning. The patient‑level smoothing is particularly valuable for real‑world screening where multiple views of the same lesion are common. Limitations include reliance on synthetic HSI whose fidelity may differ from true hyperspectral measurements, potential sensitivity to the choice of α and the number of base learners (not extensively ablated), and the lack of external validation on datasets from other institutions.

Conclusion
The proposed patient‑aware multimodal RGB‑HSI fusion with Incremental Heuristic Meta‑Learning delivers robust oral lesion classification despite small, imbalanced, and heterogeneous data. It demonstrates that spectral reconstruction from RGB, together with calibrated ensemble learning and patient‑level posterior smoothing, can substantially improve diagnostic reliability in low‑resource oral cancer screening scenarios. Future work should explore true hyperspectral acquisition, cross‑site generalization, and automated hyperparameter optimization to further solidify the approach for clinical deployment.