MePo: Meta Post-Refinement for Rehearsal-Free General Continual Learning

To cope with uncertain changes of the external world, intelligent systems must continually learn from complex, evolving environments and respond in real time. This ability, collectively known as general continual learning (GCL), encapsulates practical challenges such as online datastreams and blurry task boundaries. Although leveraging pretrained models (PTMs) has greatly advanced conventional continual learning (CL), these methods remain limited in reconciling the diverse and temporally mixed information along a single pass, resulting in sub-optimal GCL performance. Inspired by meta-plasticity and reconstructive memory in neuroscience, we introduce here an innovative approach named Meta Post-Refinement (MePo) for PTMs-based GCL. This approach constructs pseudo task sequences from pretraining data and develops a bi-level meta-learning paradigm to refine the pretrained backbone, which serves as a prolonged pretraining phase but greatly facilitates rapid adaptation of representation learning to downstream GCL tasks. MePo further initializes a meta covariance matrix as the reference geometry of pretrained representation space, enabling GCL to exploit second-order statistics for robust output alignment. MePo serves as a plug-in strategy that achieves significant performance gains across a variety of GCL benchmarks and pretrained checkpoints in a rehearsal-free manner (e.g., 15.10%, 13.36%, and 12.56% on CIFAR-100, ImageNet-R, and CUB-200 under Sup-21/1K). Our source code is available at \href{https://github.com/SunGL001/MePo}{MePo}

💡 Research Summary

The paper addresses the challenge of General Continual Learning (GCL), where an AI system must learn from a single-pass online data stream and cope with blurry task boundaries that cause label overlap and the absence of a test‑time task oracle. While pretrained models (PTMs) have boosted conventional continual learning (CL), existing PTM‑based CL methods still struggle under GCL conditions, especially when self‑supervised PTMs are used, because they cannot reconcile temporally mixed information in a single pass and rely on replay or explicit task boundaries.

To overcome these limitations, the authors propose Meta Post‑Refinement (MePo), a plug‑in framework that prepares PTMs for GCL before the actual continual learning phase. MePo consists of two complementary components: (1) a meta‑learning based representation refinement stage and (2) a meta‑covariance matrix for robust output alignment.

In the representation refinement stage, MePo constructs pseudo‑task sequences from the original pretraining dataset. For each meta‑epoch, a random subset of classes and a fixed number of samples per class are sampled, then split into a meta‑training set and a meta‑validation set. The meta‑training set is further divided into a sequence of pseudo‑tasks that mimic the sequential nature of GCL. A bi‑level optimization is performed: the inner loop sequentially updates the backbone parameters θ and the classifier parameters ψ on each pseudo‑task using standard cross‑entropy loss, thereby simulating online learning; the outer loop jointly fine‑tunes the backbone on the held‑out meta‑validation set to correct any bias introduced by the sequential inner updates. After the outer loop, a first‑order meta‑update (θ←θ+η_meta·Δθ) moves the backbone toward a representation that can quickly adapt to any future task sequence. This meta‑refinement is performed once, before deployment, and incurs no additional cost during the actual GCL phase.

The second component, the meta‑covariance matrix Σ_meta, is initialized from the second‑order statistics of the pretrained feature space. During GCL, incoming samples are projected onto this reference geometry, effectively normalizing and reconstructing their features. This alignment mechanism replaces heuristic logit‑masking and provides stable output predictions even when class labels overlap across tasks.

The authors evaluate MePo on three widely used PTM checkpoints—Supervised ImageNet‑21K (Sup‑21K), a self‑supervised variant fine‑tuned on ImageNet‑1K (Sup‑21/1K), and the self‑supervised iBOT‑21K—and on three GCL benchmarks: CIFAR‑100, ImageNet‑R, and CUB‑200 under the Si‑Blurry setting (which introduces both online streaming and blurry boundaries). MePo consistently outperforms state‑of‑the‑art PTM‑based CL/GCL methods such as L2P, DualPrompt, MVP, and MISA, achieving absolute accuracy gains of 12.56 % to 15.10 % across datasets. Notably, the gains are larger for self‑supervised PTMs, where prior methods suffer severe performance drops.

Ablation studies dissect the contributions of each module. Removing the meta‑update (i.e., using only prompt tuning) leads to a substantial drop in performance, confirming the importance of backbone refinement. Excluding the meta‑covariance while retaining a conventional logit mask results in instability under blurry task boundaries, demonstrating that Σ_meta is essential for robust output alignment.

Overall, MePo introduces a novel “pretraining → meta post‑refinement → GCL” pipeline that equips pretrained models with a meta‑learned inductive bias and a geometrically grounded alignment mechanism. It eliminates the need for replay buffers, reduces memory and privacy concerns, and enables rapid, rehearsal‑free adaptation to realistic continual learning scenarios. The work thus paves the way for more practical deployment of large pretrained models in dynamic, real‑world environments.