Towards Metamerism via Foveated Style Transfer

The problem of $\textit{visual metamerism}$ is defined as finding a family of perceptually indistinguishable, yet physically different images. In this paper, we propose our NeuroFovea metamer model, a foveated generative model that is based on a mixture of peripheral representations and style transfer forward-pass algorithms. Our gradient-descent free model is parametrized by a foveated VGG19 encoder-decoder which allows us to encode images in high dimensional space and interpolate between the content and texture information with adaptive instance normalization anywhere in the visual field. Our contributions include: 1) A framework for computing metamers that resembles a noisy communication system via a foveated feed-forward encoder-decoder network – We observe that metamerism arises as a byproduct of noisy perturbations that partially lie in the perceptual null space; 2) A perceptual optimization scheme as a solution to the hyperparametric nature of our metamer model that requires tuning of the image-texture tradeoff coefficients everywhere in the visual field which are a consequence of internal noise; 3) An ABX psychophysical evaluation of our metamers where we also find that the rate of growth of the receptive fields in our model match V1 for reference metamers and V2 between synthesized samples. Our model also renders metamers at roughly a second, presenting a $\times1000$ speed-up compared to the previous work, which allows for tractable data-driven metamer experiments.

💡 Research Summary

The paper tackles the long‑standing problem of visual metamer generation—producing physically different images that are perceptually indistinguishable—by introducing a feed‑forward, foveated style‑transfer architecture called NeuroFovea. Traditional metamer synthesis, exemplified by Freeman & Simoncelli (2011), relies on iterative gradient‑descent to match local texture statistics (Portilla & Simoncelli, 2000) across log‑polar pooling regions that mimic V1/V2 receptive fields. This process is computationally prohibitive, taking several hours for a grayscale 512 × 512 image and up to a day for color, which prevents large‑scale psychophysical studies and real‑time applications such as gaze‑contingent displays.

NeuroFovea replaces the iterative pipeline with a single forward pass using a VGG‑19 encoder‑decoder pair augmented by Adaptive Instance Normalization (AdaIN). The input image I is encoded to a high‑dimensional feature map C = E(I). Simultaneously, a noise patch N, drawn from a ZCA‑whitened distribution matching the mean and variance of I, is encoded to N = E(N). AdaIN then stylizes the noise by aligning its channel‑wise statistics to those of C, yielding S(N). For each foveated pooling region i, a spatial mask w_i and a content‑texture trade‑off coefficient α_i are defined. The target feature for region i is a convex combination: T_i = (1 − α_i)·C_i + α_i·S(N_i). All region‑wise targets are summed using the masks to form a global target T = Σ_i w_i·T_i, which is decoded by D to produce the metamer M = D(T). A pix2pix‑U‑Net refinement module is attached to the decoder to improve high‑resolution fidelity.

The key theoretical insight is that perturbations in the encoded space that lie in the perceptual null space (the component orthogonal to the human visual system’s projection) are invisible to observers. By decomposing the difference S(N_i) − C_i into a null component and a perceptually relevant component, the authors show that α_i can be increased up to a critical value without breaking metamerism, provided the perceptually relevant component remains below a detection threshold. This explains why texture‑rich regions tolerate larger α_i values, while texture‑poor regions require smaller α_i.

A major challenge is the hyper‑parameter explosion: unlike the earlier model, NeuroFovea must specify α_i for every pooling region. The authors introduce a function γ(·; s) that maps eccentricity‑dependent receptive‑field scale s to the appropriate α_i, effectively collapsing the multi‑dimensional search into a single scalar optimization over s. Experiment 1 uses a computational simulation to estimate γ, demonstrating that a relationship between α and receptive‑field size exists. Experiment 2 conducts a human ABX task across several scales. Results show that when s matches V1‑level receptive fields, observers cannot distinguish the synthesized metamer from the reference (metameric condition). When s corresponds to V2‑level fields, discrimination improves, confirming that the model’s receptive‑field growth aligns with neurophysiological data.

The contributions are fourfold: (1) a foveated, feed‑forward style‑transfer framework for metamer synthesis that eliminates gradient descent; (2) an explicit modeling of perceptual null‑space perturbations via stylized noise; (3) a principled reduction of hyper‑parameters through the γ function; and (4) a 1000× speedup (≈1 s per image), enabling large‑scale psychophysics and real‑time applications such as VR/AR gaze‑contingent rendering or metameric video for neuroimaging. The work bridges information‑theoretic concepts (noisy communication channels) with visual neuroscience, suggesting future directions in compression, rate‑distortion theory, and adaptive display technologies.