Semantic learning in autonomously active recurrent neural networks

The human brain is autonomously active, being characterized by a self-sustained neural activity which would be present even in the absence of external sensory stimuli. Here we study the interrelation between the self-sustained activity in autonomously active recurrent neural nets and external sensory stimuli. There is no a priori semantical relation between the influx of external stimuli and the patterns generated internally by the autonomous and ongoing brain dynamics. The question then arises when and how are semantic correlations between internal and external dynamical processes learned and built up? We study this problem within the paradigm of transient state dynamics for the neural activity in recurrent neural nets, i.e. for an autonomous neural activity characterized by an infinite time-series of transiently stable attractor states. We propose that external stimuli will be relevant during the sensitive periods, {\it viz} the transition period between one transient state and the subsequent semi-stable attractor. A diffusive learning signal is generated unsupervised whenever the stimulus influences the internal dynamics qualitatively. For testing we have presented to the model system stimuli corresponding to the bars and stripes problem. We found that the system performs a non-linear independent component analysis on its own, being continuously and autonomously active. This emergent cognitive capability results here from a general principle for the neural dynamics, the competition between neural ensembles.

💡 Research Summary

The paper investigates how semantic relationships between internally generated neural activity and external sensory input can emerge in an autonomously active recurrent neural network (RNN). The authors begin by emphasizing that the brain exhibits continuous, self‑sustained dynamics even in the absence of stimuli, and they ask when and how such dynamics become meaningfully linked to the outside world. To address this, they adopt the framework of transient‑state dynamics: the network continuously traverses a sequence of semi‑stable attractor states, each lasting only a brief interval before the system moves to the next. The transition intervals are termed “sensitive periods.” During these periods the network is especially receptive to external perturbations; if a stimulus qualitatively changes the trajectory, a diffusive learning signal is generated automatically, without any external teacher or error signal.

The model is built from competing neural ensembles. Each ensemble can become transiently dominant, suppressing others through inhibitory interactions. This competition creates a non‑linear landscape in which the system’s trajectory is shaped by both internal dynamics and occasional external nudges. The diffusive learning signal is not a global error‑backpropagation term but a locally driven, activity‑dependent diffusion that gradually adjusts synaptic weights of the active ensemble and its neighbors. The authors argue that this mechanism mirrors biological neuromodulatory processes (e.g., dopamine diffusion) that gate plasticity based on salient events.

To test the theory, the authors present the classic “bars and stripes” problem, a benchmark for independent component analysis (ICA). The task requires the network to separate horizontal and vertical bar patterns that are superimposed in a set of binary images. In the simulation, the RNN runs autonomously, producing a stream of transient attractors. When a bar pattern is presented during a sensitive period, the diffusive signal modifies the weights so that a specific attractor becomes associated with that pattern. Over time, the network self‑organizes a set of attractors that each encode one of the independent components (horizontal bar, vertical bar, or background). Remarkably, the system performs a non‑linear ICA without any explicit cost function, supervised label, or external training phase; learning emerges from the interaction of autonomous dynamics, competition, and the diffusive signal.

Key insights from the study are:

1. Sensitive periods as windows for semantic binding. The transition between attractors provides a natural moment when external input can reshape internal trajectories, allowing the formation of meaningful associations.

2. Diffusive, unsupervised learning. The learning signal propagates locally and is triggered only when the stimulus changes the qualitative course of the dynamics, offering a biologically plausible alternative to error‑driven back‑propagation.

3. Competition‑driven emergence of ICA. The rivalry among neural ensembles creates a non‑linear selection pressure that isolates statistically independent features of the input, effectively performing ICA as a by‑product of the network’s intrinsic dynamics.

The authors conclude that autonomous, transient‑state activity combined with competition and a simple diffusive plasticity rule can give rise to sophisticated cognitive functions such as feature extraction and independent component analysis. This challenges the dominant view that learning requires explicit external supervision or a static objective function. Moreover, the framework suggests a route toward continual, energy‑efficient learning in artificial systems, where a network remains active and ready to incorporate new information at any moment, much like the brain. Future work is proposed to extend the model to richer sensory modalities, multi‑modal integration, and direct comparison with neurophysiological data, thereby testing the generality of the sensitive‑period learning principle.