Mitigating the Likelihood Paradox in Flow-based OOD Detection via Entropy Manipulation

Deep generative models that can tractably compute input likelihoods, including normalizing flows, often assign unexpectedly high likelihoods to out-of-distribution (OOD) inputs. We mitigate this likelihood paradox by manipulating input entropy based on semantic similarity, applying stronger perturbations to inputs that are less similar to an in-distribution memory bank. We provide a theoretical analysis showing that entropy control increases the expected log-likelihood gap between in-distribution and OOD samples in favor of the in-distribution, and we explain why the procedure works without any additional training of the density model. We then evaluate our method against likelihood-based OOD detectors on standard benchmarks and find consistent AUROC improvements over baselines, supporting our explanation.

💡 Research Summary

This paper tackles the well‑known “likelihood paradox” of normalizing‑flow (NF) density models, wherein out‑of‑distribution (OOD) inputs can receive higher likelihood scores than in‑distribution (ID) data. Building on the decomposition introduced by Caterini & Loaiza‑Ganem (2022), the authors note that the expected log‑likelihood gap between ID and OOD consists of a KL‑divergence term and an entropy difference term (H(Q)-H(P)). When the entropy of the OOD distribution (Q) is lower than that of the ID distribution (P), the entropy term can dominate and reverse the sign of the gap, causing the paradox.

The core idea is to increase the entropy of test inputs in a controlled manner so that the expected log‑likelihood gap aligns with intuition (ID > OOD). The authors first provide a theoretical analysis: by adding isotropic Gaussian noise (Z\sim\mathcal N(0,\sigma^{2}I)) to a sample (X\sim Q), the resulting distribution (Q’) has higher entropy. Theorem 3.1 proves a lower bound on the expected log‑likelihood difference after perturbation, showing that the bound grows monotonically with the noise variance (\sigma^{2}). This establishes that entropy augmentation can systematically improve OOD detection for any NF model without retraining.

Since at inference time we cannot know whether a sample is ID or OOD, the authors devise a practical, training‑free scheme called Semantic Proportional Entropy Manipulation (SPEM). SPEM uses a pretrained feature extractor (g(\cdot)) (e.g., a ResNet trained on ImageNet) to embed each ID training sample into a memory bank (M). For a test image (x_{\text{test}}), its embedding (h_{\text{test}}) is compared to the bank via cosine similarity; the maximum similarity (\lambda\in