Beyond Grid-Locked Voxels: Neural Response Functions for Continuous Brain Encoding

Neural encoding models aim to predict fMRI-measured brain responses to natural images. fMRI data is acquired as a 3D volume of voxels, where each voxel has a defined spatial location in the brain. However, conventional encoding models often flatten this volume into a 1D vector and treat voxel responses as independent outputs. This removes spatial context, discards anatomical information, and ties each model to a subject-specific voxel grid. We introduce the Neural Response Function (NRF), a framework that models fMRI activity as a continuous function over anatomical space rather than a flat vector of voxels. NRF represents brain activity as a continuous implicit function: given an image and a spatial coordinate (x, y, z) in standardized MNI space, the model predicts the response at that location. This formulation decouples predictions from the training grid, supports querying at arbitrary spatial resolutions, and enables resolution-agnostic analyses. By grounding the model in anatomical space, NRF exploits two key properties of brain responses: (1) local smoothness – neighboring voxels exhibit similar response patterns; modeling responses continuously captures these correlations and improves data efficiency, and (2) cross-subject alignment – MNI coordinates unify data across individuals, allowing a model pretrained on one subject to be fine-tuned on new subjects. In experiments, NRF outperformed baseline models in both intrasubject encoding and cross-subject adaptation, achieving high performance while reducing the data size needed by orders of magnitude. To our knowledge, NRF is the first anatomically aware encoding model to move beyond flattened voxels, learning a continuous mapping from images to brain responses in 3D space.

💡 Research Summary

The paper introduces Neural Response Function (NRF), a novel framework for fMRI visual encoding that treats brain responses as a continuous function over standardized anatomical space rather than as a flattened vector of voxels. Traditional encoding models flatten the 3‑D fMRI volume into a 1‑D response vector, ignoring spatial relationships between neighboring voxels and binding each model to a subject‑specific voxel grid. This leads to poor data efficiency and prevents transfer across subjects.

NRF addresses these limitations by conditioning predictions on both the visual stimulus and the voxel’s MNI coordinates. The model consists of two components: (1) an image feature extractor G(M) that produces multi‑scale embeddings of the stimulus, and (2) an implicit neural representation predictor P that receives the concatenated image embedding and a Fourier‑encoded coordinate γ(x). The final prediction is Φ(M, x) = P(G(M), γ(x)), yielding a scalar response for any point x ∈ ℝ³. By using Fourier features for the coordinates, the network can capture high‑frequency spatial variations while preserving smoothness across nearby locations.

Training is performed end‑to‑end with Adam (lr = 3e‑3). Each mini‑batch samples 32 images and, for each image, 2 000 voxels (out of ~13–15 k). The loss combines mean‑squared error and cosine similarity (α = 0.1), encouraging both accurate magnitude and correct directional alignment of predicted response vectors. This dual objective helps the model learn the fine‑grained pattern of activation across the cortex.

A key contribution is the cross‑subject transfer strategy. Because all responses are defined in MNI space, a model pretrained on one or several subjects can be fine‑tuned on a new individual using only a few hundred stimulus‑response pairs. Both the visual encoder G and the coordinate‑conditioned predictor P are updated, allowing the model to adapt to individual differences in visual processing and anatomical variability. To further boost performance, the authors ensemble K fine‑tuned models voxel‑wise: for each voxel v they learn linear weights w_{v,k} and a bias b_v that combine the K predictions, solving a simple least‑squares problem on the limited new‑subject data.

Empirical results show that NRF outperforms conventional voxel‑wise linear and deep models on two fronts. First, within a single subject, NRF achieves higher Pearson correlation (≈10‑15 % improvement) and produces smoother activation maps, especially in visual areas such as V1‑V4, fusiform face area, and extrastriate body area. Second, in low‑data regimes (200–300 training images), NRF retains most of its performance, whereas discrete models degrade sharply. When transferred to a new subject, NRF with fine‑tuning and voxel‑wise ensembling reaches twice the predictive accuracy of baseline models trained from scratch on the same limited data.

The paper also discusses limitations. The current evaluation focuses mainly on visual cortex; generalization to higher‑order regions remains to be demonstrated. Sensitivity to the number of Fourier frequencies and MLP depth is not exhaustively explored, and the method’s computational cost for real‑time inference is not addressed. Future work could extend NRF to multimodal stimuli, incorporate attention mechanisms, or develop gradient‑based saliency maps that visualize the learned spatial fields, thereby enhancing neuroscientific interpretability.

In summary, NRF provides a paradigm shift for fMRI encoding: by modeling brain activity as a continuous, anatomically grounded function, it leverages local spatial smoothness and cross‑subject anatomical alignment to achieve superior data efficiency and transferability. This approach opens the door to more flexible, resolution‑agnostic brain models and has potential impact on brain‑computer interfaces, clinical neuroimaging, and the construction of functional digital twins of the human visual system.