Recurrent neural networks (RNNs) provide a theoretical framework for understanding computation in biological neural circuits, yet classical results, such as Hopfield's model of associative memory, rely on symmetric connectivity that restricts network dynamics to gradient-like flows. In contrast, biological networks support rich time-dependent behaviour facilitated by their asymmetry. Here we introduce a general framework, which we term drift-diffusion matching, for training continuous-time RNNs to represent arbitrary stochastic dynamical systems within a low-dimensional latent subspace. Allowing asymmetric connectivity, we show that RNNs can faithfully embed the drift and diffusion of a given stochastic differential equation, including nonlinear and nonequilibrium dynamics such as chaotic attractors. As an application, we construct RNN realisations of stochastic systems that transiently explore various attractors through both input-driven switching and autonomous transitions driven by nonequilibrium currents, which we interpret as models of associative and sequential (episodic) memory. To elucidate how these dynamics are encoded in the network, we introduce decompositions of the RNN based on its asymmetric connectivity and its time-irreversibility. Our results extend attractor neural network theory beyond equilibrium, showing that asymmetric neural populations can implement a broad class of dynamical computations within low-dimensional manifolds, unifying ideas from associative memory, nonequilibrium statistical mechanics, and neural computation.
Understanding how populations of neurons encode information and perform computations is one of the most significant problems in neuroscience [1]. Central to the 'connectionist' perspective, is the idea that cognition is performed by distributed networks of interacting neurons. However, unlike the feed-forward architectures that dominate modern machine learning, the neuronal networks of the brain are recurrent, leading to a rich repertoire of time-evolving dynamics. As a result, classes of recurrent neural networks (RNNs) have become the prototypical models for dynamic computation in biological neural circuits [2]. These networks, and variants thereof, have been used extensively to model neural data [3,4], develop theories of neural computation [5,6], as well as perform sequence learning tasks, such as the prediction and encoding of chaotic time-series [7][8][9].
One of the most influential results in the study of RNNs is Hopfield’s model of associative memory [10,11]. In the continuous version of the model, the dynamics are identical to that of a so-called ‘vanilla’ RNN. Such a system is composed of n neurons, each with internal state u i and output state v i = h(u i ), where h is a monotonic nonlinear activation function (we will take h(x) = tanh(x)). The dynamics of the internal state are given by the stochastic differential equation (SDE),
where u = (u 1 , .., u n ) and F = (F 1 , …, F n ) is the drift function, with,
Here B ∈ R n×d and w(t) is a Wiener process with d independent sources of noise [12]. We also define ϒ = BB ⊤ /2 to be the positive semidefinite diffusion matrix of rank d. The connectivity from neuron j to neuron i is given by W i j , and I i is the input current to neuron i. When the connectivity is constrained to be symmetric, W = W ⊤ , we introduce the energy potential,
and Ẽ(u) = E(h(u)). The drift can be written in the form,
= -D(u)∇ u Ẽ(u), (5) where
is a positive-definite matrix-valued function. As a result, the RNN is constrained to perform ‘generalised’ gradient descent on the energy landscape, eventually arriving at an energy minimum. 1 Hopfield’s model allows for the manipulation of the energy landscape via training, such that a desired ‘memory’ can be encoded as an energy minimum. The result is an autonomous neural circuit that will ‘retrieve the memory given initial information’ i.e. converge to the energy minimum from an appropriate initial state. Hopfield’s results laid the ground work for the study of attractor neural networks, and their possible implications for neuroscience [13,14]. Despite Hopfield’s application to associative memory, limiting to symmetric connectivity severely constrains the repertoire of dynamics within the system, precluding the existence of limit-cycles and other dynamic phenomena. Moreover, asymmetry is ubiquitous in biological neural circuits, thus the symmetric assumption limits the neuroscientific plausibility of such a model. Numerous studies have analysed asymmetric Hopfield networks, but these typically consider the discrete, spin-glass form of the model [15][16][17], and analyse the attracting states emerging from random connectivity. On the other hand, in the continuous model, asymmetric connectivity disrupts the energy landscape eroding the ability of the network to preserve attracting states [18]. Moreover, the dichotomy of symmetric and asymmetric connectivity is closely related to the necessary departure from equilibrium to nonequilibrium statistical mechanics, which has become an emerging area of research in neuroscience, neural computation, and associative memory [18][19][20][21][22][23][24]. 2In an entirely separate development, dimensionality reduction techniques have shown the emergence of lowdimensional, latent structure in high-dimensional neural activity associated with tasks like motor control, navigation and working memory [25,26]. These so-called neural manifolds indicate that high-dimensional brain network activity is, in fact, constrained to low-dimensional subspaces where the dynamics are often more mechanistically interpretable. These low-dimensional dynamics are responsible for computation and may also emerge in RNNs [27], typically associated with low-rank structure in the connectivity [28,29]. Neural manifolds are just a single example forming part of the more general hypothesis that real-world network dynamics are effectively low-dimensional [30,31], which in turn can be seen as a special case of the well-known manifold hypothesis for highdimensional data [32]. Neural data analysis has provided significant evidence that computational dynamics are encoded at the level of latent manifolds in empiricial brain network activity [25,26,33]. As a result, understanding how such cognitive processes can be encoded in RNNs via neural manifolds is crucial to understanding the mechanisms and limits of neural computation.
In this article, we fuse together asymmetric attractor neural networks with the theory of neural manifo
This content is AI-processed based on open access ArXiv data.