Generative Score Inference for Multimodal Data

Accurate uncertainty quantification is crucial for making reliable decisions in various supervised learning scenarios, particularly when dealing with complex, multimodal data such as images and text. Current approaches often face notable limitations, including rigid assumptions and limited generalizability, constraining their effectiveness across diverse supervised learning tasks. To overcome these limitations, we introduce Generative Score Inference (GSI), a flexible inference framework capable of constructing statistically valid and informative prediction and confidence sets across a wide range of multimodal learning problems. GSI utilizes synthetic samples generated by deep generative models to approximate conditional score distributions, facilitating precise uncertainty quantification without imposing restrictive assumptions about the data or tasks. We empirically validate GSI’s capabilities through two representative scenarios: hallucination detection in large language models and uncertainty estimation in image captioning. Our method achieves state-of-the-art performance in hallucination detection and robust predictive uncertainty in image captioning, and its performance is positively influenced by the quality of the underlying generative model. These findings underscore the potential of GSI as a versatile inference framework, significantly enhancing uncertainty quantification and trustworthiness in multimodal learning.

💡 Research Summary

The paper introduces Generative Score Inference (GSI), a unified framework for uncertainty quantification in multimodal supervised learning tasks such as image‑text pairs, tabular data, and large language model (LLM) outputs. The central idea is to define a task‑specific score function s(y, ŷ) that measures the discrepancy between a model’s prediction ŷ and the true response y (e.g., L1 loss for regression, cross‑entropy or ROUGE‑L for text, aggregate loss for captioning). Rather than relying on marginal conformal prediction, GSI seeks conditional coverage: for each input x it constructs a prediction set Cα(x) that contains the true y with probability at least 1 − α given x.

To achieve this, GSI learns the conditional distribution of the score, P(s | x), using a deep generative model. The authors primarily employ conditional diffusion models because they can represent highly non‑Gaussian, multimodal score distributions and are relatively stable to train. After fitting the generator on a calibration set of (x, s) pairs, the model draws a large synthetic sample {˜s_j} for a new x, computes the empirical (1 − α) quantile q̂1‑α, and defines the prediction set as all y whose score does not exceed this quantile. Theoretical analysis introduces a “generation error” assumption: the total‑variation distance between the true score distribution and the learned one is bounded by τ with high probability. Under this assumption, the paper proves asymptotic conditional coverage and provides explicit error bounds that vanish as the calibration size grows and the generator improves.

The framework is evaluated on three representative domains. In standard tabular regression benchmarks, GSI produces narrower intervals than state‑of‑the‑art conformal methods while maintaining ≥90 % coverage. For hallucination detection in LLM question‑answering, the authors define a semantic‑dissimilarity score and show that GSI‑based hypothesis testing outperforms recent Semantic Entropy and LLM‑Check approaches in both precision and recall. In image caption selection on MS‑COCO, GSI identifies images that a vision‑language model can caption reliably, achieving higher statistical power and better selection accuracy than the recent Conformal Alignment method. Across all experiments, performance improves as the underlying diffusion model’s sampling steps increase, confirming the theoretical claim that better generative modeling yields tighter, more reliable prediction sets.

The authors discuss several limitations. GSI requires a sizable calibration set and substantial compute to train conditional generative models, which may be prohibitive in low‑resource settings. Designing an appropriate score function still demands domain expertise, and the conditional coverage guarantee hinges on the generator’s ability to approximate P(s | x) closely; large TV distance can degrade the guarantee. Future work suggested includes more efficient sampling (e.g., importance or variational techniques), learning score functions automatically via meta‑learning, extending the theoretical analysis to other generator families such as normalizing flows or GANs, and developing lightweight, online versions suitable for real‑time inference in deployed LLM services.

In summary, GSI offers a theoretically sound, flexible, and empirically strong solution for uncertainty quantification across heterogeneous multimodal tasks. By leveraging modern generative models to estimate conditional score distributions, it overcomes the restrictive assumptions of classical conformal prediction and delivers tighter, better‑calibrated prediction sets. As generative modeling continues to advance, GSI’s applicability and performance are poised to grow, making it a promising tool for building trustworthy AI systems in domains where reliable uncertainty estimates are essential.