A Proper Scoring Rule for Virtual Staining

Generative virtual staining (VS) models for high-throughput screening (HTS) can provide an estimated posterior distribution of possible biological feature values for each input and cell. However, when evaluating a VS model, the true posterior is unavailable. Existing evaluation protocols only check the accuracy of the marginal distribution over the dataset rather than the predicted posteriors. We introduce information gain (IG) as a cell-wise evaluation framework that enables direct assessment of predicted posteriors. IG is a strictly proper scoring rule and comes with a sound theoretical motivation allowing for interpretability, and for comparing results across models and features. We evaluate diffusion- and GAN-based models on an extensive HTS dataset using IG and other metrics and show that IG can reveal substantial performance differences other metrics cannot.

💡 Research Summary

High‑throughput screening (HTS) relies on fluorescence microscopy to assess thousands of compounds in parallel. Traditional virtual staining (VS) pipelines translate bright‑field images into synthetic fluorescence images using either deterministic regression or conditional generative models (GANs, diffusion models). While generative VS models can produce many samples per input, allowing an approximate posterior distribution of cell‑level biological features Y, the true posterior P(Y|x) remains inaccessible because only a single fluorescence image (and thus a single feature value) is observed for each bright‑field input.

Existing evaluation protocols sidestep this problem by comparing the marginal distribution of features across the whole dataset, typically using Kullback‑Leibler divergence (KLD) or image‑level metrics such as Inception Score or FID. These approaches ignore whether a model correctly captures the conditional uncertainty for each individual cell, which is crucial when the goal is to infer biologically meaningful information from a single bright‑field snapshot.

The authors propose three evaluation metrics that operate at the cell level:

1. Marginal KLD – the classic KL divergence between the empirical marginal feature distribution P(Y) and the model‑generated marginal Pθ(Y). This metric does not require knowledge of the true posterior but only assesses overall distributional similarity.

2. Rank Metric (Probability Integral Transform) – for each cell, the true feature value Y is ranked among K samples drawn from the model’s posterior Pθ(Y|x). If the model is perfect, the collection of ranks should be uniformly distributed. Deviation from uniformity is quantified by the 1‑Wasserstein distance (Rank Distance). Although intuitive, this metric is not a strictly proper scoring rule; it can mask systematic mis‑calibration of the predicted density.

3. Information Gain (IG) – the core contribution. The average log‑likelihood of the true values under the model posterior, ℓ̄θ = (1/N)∑log Pθ(Y|x), is computed and then compared to the log‑likelihood under a reference model that ignores the bright‑field input (the marginal P(Y)). The difference, IG = ℓ̄θ − ℓ̄ref, equals the average reduction in KL divergence achieved by conditioning on x. Because the log‑likelihood is the logarithmic score, IG is a strictly proper scoring rule: it is maximized only when the predicted posterior matches the true posterior. Moreover, the constant offset that prevents direct comparison of raw log‑likelihoods cancels out, enabling fair comparison across features and datasets.

To validate these metrics, the authors trained two representative VS models on a large HTS dataset consisting of 30 000 paired bright‑field and DAPI images from six cell lines, 49 plates, and ten toxic compounds. The models are:

• Pix2PixHD GAN – a multi‑scale conditional GAN with three discriminators, employing MC‑Dropout at inference to generate stochastic samples.
• Conditional Denoising Diffusion Probabilistic Model (cDDPM) – a diffusion‑based generator that receives the bright‑field image as an additional channel.

Both models were trained for up to 200 epochs with early stopping, and during inference 1 000 virtual fluorescence samples per bright‑field image were generated. Sampled images were processed through a Cellpose‑based nuclei segmentation pipeline, yielding 18 quantitative cell features (seven intensity‑based, eleven radial distribution metrics).

Qualitative observations (Figure 2) show that cDDPM reproduces cell shape, size, and intensity profiles more faithfully than the GAN, and its samples exhibit substantially higher variability, reflecting a broader posterior.

Quantitative evaluation reveals striking differences among the three metrics:

• Marginal KLD – both models achieve nearly identical values (~0.08), suggesting they capture the overall feature distribution equally well.
• Rank Distance – also similar for the two models (≈0.09), indicating that the uniformity of ranks does not differentiate their conditional performance.
• Information Gain – cDDPM outperforms Pix2PixHD by an average log‑likelihood increase of 10.54 points, translating into a markedly higher IG across all 18 features. The log‑likelihood distribution for the GAN displays a long left tail, meaning many cells receive very low probability under the predicted posterior, whereas cDDPM’s distribution is tightly centered.

When aggregating results across all features (Figure 4), IG consistently ranks cDDPM above the GAN, with especially large gaps for intensity‑related features (F1, F3, F4, F7). In contrast, marginal KLD and rank distance fail to reveal any systematic advantage, sometimes even suggesting the opposite. The authors argue that because IG is a strictly proper scoring rule, it reflects genuine differences in posterior quality, while the rank metric’s lack of properness can hide such discrepancies.

Implications:

1. Evaluation Paradigm Shift – Relying solely on marginal statistics can be misleading for generative VS models; cell‑wise proper scoring rules like IG provide a more faithful assessment of conditional information extraction.
2. Model Selection for HTS – In drug discovery pipelines where each bright‑field image is costly to acquire, a model with higher IG promises more reliable inference of biologically relevant features from fewer experiments.
3. Generalizability – The IG framework is applicable beyond VS, to any setting where a model predicts a conditional distribution from limited observations (e.g., single‑cell RNA‑seq imputation, pharmacodynamic response prediction).

In summary, the paper introduces Information Gain as a theoretically sound, strictly proper scoring rule for evaluating virtual staining models at the cellular level. Through extensive experiments on a realistic HTS dataset, the authors demonstrate that IG uncovers performance gaps missed by conventional marginal‑level metrics, thereby offering a robust tool for benchmarking and improving generative models in high‑throughput biomedical imaging.