How 'Neural' is a Neural Foundation Model?

Foundation models have shown remarkable success in fitting biological visual systems; however, their black-box nature inherently limits their utility for understanding brain function. Here, we peek inside a SOTA foundation model of neural activity (Wang et al., 2025) as a physiologist might, characterizing each ’neuron’ based on its temporal response properties to parametric stimuli. We analyze how different stimuli are represented in neural activity space by building decoding manifolds, and we analyze how different neurons are represented in stimulus-response space by building neural encoding manifolds. We find that the different processing stages of the model (i.e., the feedforward encoder, recurrent, and readout modules) each exhibit qualitatively different representational structures in these manifolds. The recurrent module shows a jump in capabilities over the encoder module by ‘pushing apart’ the representations of different temporal stimulus patterns. Our ’tubularity’ metric quantifies this stimulus-dependent development of neural activity as biologically plausible. The readout module achieves high fidelity by using numerous specialized feature maps rather than biologically plausible mechanisms. Overall, this study provides a window into the inner workings of a prominent neural foundation model, gaining insights into the biological relevance of its internals through the novel analysis of its neurons’ joint temporal response patterns. Our findings suggest design changes that could bring neural foundation models into closer alignment with biological systems: introducing recurrence in early encoder stages, and constraining features in the readout module.

💡 Research Summary

This paper conducts a thorough neuroscientific dissection of a state‑of‑the‑art foundation model of neural activity (the Foundation Neural Network, FNN) to assess how “neural” its internal representations truly are. The authors adopt three complementary analysis pipelines that are standard in systems neuroscience: (1) decoding manifolds, which embed each stimulus trial in the high‑dimensional space of neural responses and then reduce dimensionality (via PCA) to visualize stimulus separability; (2) encoding manifolds, which treat each artificial neuron as a point in a stimulus‑response coordinate system and use tensor factorization to reveal functional topologies; and (3) temporal trajectories (PSTHs and streamline traces) that capture the dynamical evolution of neural activity during stimulus presentation.

The FNN, trained on the MICrONS mouse visual dataset, consists of a perspective module, a modulation module, a deep convolutional encoder (10 layers, including 3‑D convolutions for short‑term temporal integration), a Conv‑LSTM‑based recurrent module preceded by attention, and a per‑mouse readout that interpolates recurrent outputs to produce predicted neural firing rates.

Decoding‑manifold results show that biological data from mouse retina and primary visual cortex form tight, stimulus‑specific clusters, confirming that neural responses at those stages are readily readable. In contrast, the first encoder layer (L1) of the FNN yields a poorly clustered manifold with mixed stimulus classes and the lowest classification accuracy (≈0.74). Layer 8 improves modestly, but the recurrent layer achieves the most distinct clusters and the highest accuracy (≈0.89), indicating that recurrence is the key mechanism that “pushes apart” representations of different temporal patterns. The readout and output layers, despite achieving high predictive performance, display weaker clustering, suggesting they act more as fitting mechanisms than as generators of biologically plausible representations.

Encoding‑manifold analyses further differentiate the model from biology. The retinal manifold exhibits discrete clusters corresponding to known ganglion‑cell types, while V1 shows a continuous topology reflecting smooth variations in direction selectivity. The FNN encoder produces two prominent arms: an orientation‑selective arm (α) that aligns with biological findings, and an intensity‑based arm (γ) lacking a clear biological counterpart. The recurrent stage introduces many direction‑selective units, mirroring cortical dynamics, whereas the readout stage collapses into a highly clustered “bottleneck” where many units share the same feature map—a pattern that boosts accuracy but deviates from the distributed coding observed in cortex. The final output layer regains a continuous topology, yet individual unit PSTHs remain unlike those of real V1 neurons.

To quantify the temporal structuring introduced by recurrence, the authors introduce a “tubularity” metric, which rises sharply at the recurrent stage, confirming that this module creates stimulus‑dependent, tube‑like trajectories in neural state space.

Overall, the study reveals a mismatch between the FNN’s early feed‑forward encoder and the biological retina, a beneficial but late emergence of temporal separation in the recurrent module, and an over‑specialized readout that sacrifices biological realism for predictive power. The authors propose two design modifications: (1) embed recurrence already in the early encoder layers to integrate temporal information sooner, and (2) constrain or regularize the readout weights to promote a more distributed, biologically plausible feature representation. Implementing these changes could bring foundation models closer to true “digital twins” of the visual brain, enhancing their utility for hypothesis testing and neuroscientific discovery.