Do Neural Networks Lose Plasticity in a Gradually Changing World?

Continual learning has become a trending topic in machine learning. Recent studies have discovered an interesting phenomenon called loss of plasticity, referring to neural networks gradually losing the ability to learn new tasks. However, existing plasticity research largely relies on contrived settings with abrupt task transitions, which often do not reflect real-world environments. In this paper, we propose to investigate a gradually changing environment, and we simulate this by input/output interpolation and task sampling. We perform theoretical and empirical analysis, showing that the loss of plasticity is an artifact of abrupt tasks changes in the environment and can be largely mitigated if the world changes gradually.

💡 Research Summary

The paper tackles the phenomenon of “loss of plasticity” in continual learning, a situation where neural networks gradually lose the ability to acquire new tasks after training on a sequence of tasks. While many recent works have documented this effect, they typically rely on contrived benchmarks that feature abrupt task switches—e.g., random label reassignment or pixel permutation—conditions that rarely occur in real‑world environments where data distributions evolve smoothly over time. The authors argue that the observed plasticity loss may be an artifact of these sudden transitions rather than an inherent limitation of deep networks.

To test this hypothesis, they introduce two families of techniques that simulate a gradually changing world. The first family consists of input and output interpolation. In the Random Image Labeling setting, the target one‑hot vectors are linearly blended with a uniform distribution and then with the new label distribution, controlled by an interpolation coefficient α that increases stepwise. In the Random Pixel Permuting setting, the pixel‑shuffled images of two consecutive tasks are mixed as a convex combination of their pixel values. The second family is task sampling, which does not require a one‑to‑one correspondence between samples of successive tasks. Instead, a proportion (1‑α) of the current task’s data and a proportion α of the next task’s data are merged to form an intermediate dataset. By gradually increasing α from 0 to 1, the data stream changes smoothly even when the underlying tasks are fundamentally different.

The authors provide a theoretical justification based on smoothness (β‑smoothness) and local strong convexity ((r, µ)-strongly convex) of the loss landscape. They prove that, under a sufficiently small learning‑rate bound, gradient descent initialized inside a locally convex basin will stay inside that basin throughout training and converge to the local minimizer. Consequently, if the loss function evolves gradually (as induced by interpolation or sampling), the optimizer can track the moving optimum without being trapped in a poor local minimum that would otherwise arise after an abrupt shift.

Empirically, the paper evaluates four benchmark streams: (1) Random Image Labeling, (2) Random Pixel Permuting, (3) Random Seq2Seq (synthetic character‑level translation), and (4) Bigram Cipher (a structured mapping that requires learning a modular sum over a permuted vocabulary). For each benchmark, they compare three regimes: (a) abrupt task switches (the standard setting), (b) gradual transition via interpolation/sampling, and (c) existing plasticity‑preserving methods such as weight regularization, optimizer state resets, and specialized activation functions. Results consistently show that abrupt switches cause a sharp rise in training loss and a steep drop in test accuracy after a few tasks, confirming the classic plasticity‑loss pattern. In contrast, the gradual regimes maintain low training loss across all tasks, achieve higher final accuracies (typically 5–15 % above the abrupt baseline), and match or surpass the performance of the sophisticated mitigation techniques. Notably, when interpolation is feasible (e.g., label or pixel interpolation), the simple linear blending alone is enough to keep the network plastic, suggesting that the loss landscape’s smooth evolution is the key factor.

The authors also discuss practical scenarios where abrupt changes are unavoidable, such as a robot entering a completely new physical environment. Even in these cases, the proposed task sampling strategy can be applied as a lightweight “smoothing” layer: by interleaving a small fraction of data from the new environment with the old, the system experiences a softened transition that mitigates plasticity loss without requiring architectural changes.

In summary, the paper makes three major contributions: (1) it reframes loss of plasticity as a phenomenon largely driven by the artificial abruptness of benchmark designs, (2) it introduces theoretically grounded and empirically validated interpolation and sampling mechanisms that emulate a gradually changing world, and (3) it demonstrates that these simple mechanisms can replace or complement existing complex regularization or optimizer‑reset schemes. The work encourages the continual‑learning community to adopt more realistic, smoothly evolving data streams in future evaluations and opens avenues for combining gradual‑transition techniques with meta‑learning, adaptive learning‑rate schedules, and curriculum learning for even more robust lifelong learning systems.