Retrospective Feature Estimation for Continual Learning

The intrinsic capability to continuously learn a changing data stream is a desideratum of deep neural networks (DNNs). However, current DNNs suffer from catastrophic forgetting, which interferes with remembering past knowledge. To mitigate this issue, existing Continual Learning (CL) approaches often retain exemplars for replay, regularize learning, or allocate dedicated capacity for new tasks. This paper investigates an unexplored direction for CL called Retrospective Feature Estimation (RFE). RFE learns to reverse feature changes by aligning the features from the current trained DNN backward to the feature space of the old task, where performing predictions is easier. This retrospective process utilizes a chain of small feature mapping networks called retrospector modules. Empirical experiments on several CL benchmarks, including CIFAR10, CIFAR100, and Tiny ImageNet, demonstrate the effectiveness and potential of this novel CL direction compared to existing representative CL methods, motivating further research into retrospective mechanisms as a principled alternative for mitigating catastrophic forgetting in CL. Code is available at: https://github.com/mail-research/retrospective-feature-estimation.

💡 Research Summary

The paper introduces a novel continual learning (CL) paradigm called Retrospective Feature Estimation (RFE) that tackles catastrophic forgetting by learning to map the feature representations of a newly trained model back to the feature space of the previous task. Unlike traditional CL methods that either rehearse past examples, regularize parameter updates, or expand the network architecture, RFE separates the forgetting mitigation process from the primary learning of new tasks. For each incoming task t, after the standard training of the feature extractor f_t and task‑specific classifier w_t, a lightweight auxiliary network—named a retrospector module r_t—is trained to align the current features f_t(x) with the prior task’s features f_{t‑1}(x). The alignment objective is a simple L2 feature‑estimation loss: L_FE = E_x‖r_t(f_t(x), x) – f_{t‑1}(x)‖². Crucially, the retrospector can be trained without storing any past data; it only requires the previous feature extractor f_{t‑1} and the current task’s data as a proxy for the older distribution. An optional variant (RFE‑P) stores a small subset of samples from task t‑1 to further improve the mapping, offering a trade‑off between privacy, memory, and performance.

During inference, a test sample from any earlier task passes through the most recent feature extractor f_N, and then sequentially through the chain of retrospector modules r_N, r_{N‑1}, …, r_{t+1} to reconstruct an approximation of the original representation ˆf_t(x). This reconstructed feature is fed to the corresponding classifier head w_t, yielding a prediction that reflects the knowledge of the original task. In class‑incremental scenarios, predictions from multiple reconstructed representations are averaged to produce the final output.

The authors evaluate RFE on three widely used CL benchmarks—CIFAR‑10, CIFAR‑100, and Tiny‑ImageNet—under both task‑incremental and class‑incremental settings. Baselines include rehearsal‑based methods (A‑GEM, DER), regularization‑based methods (EWC, SI), and architecture‑expansion methods (Progressive Nets, Sup‑Mask). Results show that pure RFE, which requires no memory of past samples, attains accuracy comparable to rehearsal methods and degrades far more gracefully as the number of tasks grows. The RFE‑P variant, with only a tiny memory buffer for the immediate past task, often matches or slightly exceeds the performance of rehearsal baselines while still using far less storage. Computational overhead is modest because each retrospector module contains only a few hundred parameters; the added inference latency remains acceptable even with multiple modules.

Key contributions of the work are: (1) proposing a new CL direction that “corrects” feature drift after learning rather than preventing it during training; (2) demonstrating that a simple L2 alignment loss and a lightweight mapping network are sufficient to recover past representations; (3) showing that the approach can be integrated into existing CL pipelines with minimal architectural changes and optional use of past data for further gains.

The paper also discusses limitations. The retrospector chain length grows linearly with the number of tasks, which can increase inference cost for very long task sequences. The reliance on an L2 loss may be insufficient for highly non‑linear feature transformations, potentially limiting reconstruction fidelity. Future research directions suggested include designing more compact or dynamically selected retrospector modules, exploring richer loss functions such as contrastive or adversarial objectives, and investigating mechanisms to prune or merge modules to keep inference efficient over long lifetimes.

Overall, RFE offers a practical solution for continual learning in memory‑constrained environments, providing a principled way to mitigate catastrophic forgetting without sacrificing the plasticity needed for new tasks. The authors have released their code at https://github.com/mail-research/retrospective-feature-estimation, encouraging further exploration of retrospective mechanisms as an alternative CL strategy.