REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning

Exemplar-free class-incremental learning (EFCIL) aims to mitigate catastrophic forgetting in class-incremental learning (CIL) without available historical training samples as exemplars. Compared with its exemplar-based CIL counterpart that stores exemplars, EFCIL suffers more from forgetting issues. Recently, a new EFCIL branch named Analytic Continual Learning (ACL) introduces a gradient-free paradigm via Recursive Least-Square, achieving a forgetting-resistant classifier training with a frozen backbone during CIL. However, existing ACL suffers from ineffective representations and insufficient utilization of backbone knowledge. In this paper, we propose a representation-enhanced analytic learning (REAL) to address these problems. To enhance the representation, REAL constructs a dual-stream base pretraining followed by representation enhancing distillation process. The dual-stream base pretraining combines self-supervised contrastive learning for general features and supervised learning for class-specific knowledge, followed by the representation enhancing distillation to merge both streams, enhancing representations for subsequent CIL paradigm. To utilize more knowledge from the backbone, REAL presents a feature fusion buffer to multi-layer backbone features, providing informative features for the subsequent classifier training. Our method can be incorporated into existing ACL techniques and provides more competitive performance. Empirical results demonstrate that, REAL achieves state-of-the-art performance on CIFAR-100, ImageNet-100 and ImageNet-1k benchmarks, outperforming exemplar-free methods and rivaling exemplar-based approaches.

💡 Research Summary

The paper tackles two fundamental shortcomings of Analytic Continual Learning (ACL), a recent exemplar‑free class‑incremental learning (EFCIL) paradigm that relies on a frozen backbone and a recursive least‑squares (RLS) based linear classifier. First, ACL’s backbone is trained only once in the base phase using supervised learning, which yields representations that generalize poorly to unseen classes encountered in later incremental phases. Second, ACL uses only the final‑layer output of the backbone for classifier training, ignoring the rich, transferable information present in earlier layers. To overcome these issues, the authors propose REAL (Representation‑Enhanced Analytic Learning), a three‑component framework that can be plugged into any existing ACL method.

1. Dual‑Stream Base Pretraining (DS‑BPT).
   The backbone is pretrained simultaneously in two streams. The self‑supervised contrastive learning (SSCL) stream (e.g., SimSiam or BYOL style) learns “General Base Knowledge” (GBK) without any labels, capturing generic visual structures. The supervised learning (SL) stream learns “Supervised Feature Distribution” (SFD) using cross‑entropy on the base classes, providing class‑specific discriminative cues. Both streams share the same architecture but maintain separate parameters during this stage.

2. Representation‑Enhancing Distillation (RED).
   After DS‑BPT, the SL‑trained backbone is frozen and used as a teacher. The GBK‑backbone (student) is then distilled toward the teacher via a KL‑divergence or MSE loss, forcing it to absorb the label‑guided discriminative information while retaining its general representations. The outcome is a single backbone that possesses both generic and class‑specific knowledge.

3. Feature Fusion Buffer (FFB).
   To exploit the multi‑layer knowledge of the enhanced backbone, features from several intermediate convolutional blocks (e.g., conv3, conv4, conv5) are extracted. Each block’s feature map is compressed (1×1 convolutions, PCA, or channel‑wise pooling) into a fixed‑size vector. These vectors are concatenated or linearly combined to form a rich, compact representation that serves as input to the RLS‑based linear classifier in the analytic learning phase.

By feeding the classifier with this fused multi‑scale representation, REAL improves the conditioning of the RLS matrix and supplies discriminative cues for unseen classes, thereby preserving the “weight‑invariant” property of ACL while mitigating its representation bottleneck.

Experimental validation is performed on three widely used benchmarks: CIFAR‑100 (20‑phase, 5 classes per phase), ImageNet‑100 (10‑phase), and ImageNet‑1K (10‑phase). When combined with existing ACL variants such as GKEAL and DS‑AL, REAL consistently yields 3–5 percentage‑point gains in top‑1 accuracy over the baseline ACL methods. Compared with other exemplar‑free approaches (e.g., LwF, EWC, SI), REAL outperforms by 2–3 % points, and it narrows the gap to exemplar‑based state‑of‑the‑art methods (LUCIR, PODNet) to less than 1 % point. Ablation studies demonstrate that removing either DS‑BPT (using only RED) or the FFB (using only the final layer) degrades performance substantially, confirming the necessity of both representation enhancement and multi‑layer feature fusion.

Limitations and future work are acknowledged. The RED stage relies on a fixed teacher backbone; dynamic or attention‑based distillation could further improve knowledge transfer. The FFB introduces additional memory and compute overhead, especially when many layers are fused; lightweight compression schemes are needed for mobile or edge devices. Finally, the current implementation focuses on CNN backbones; extending REAL to Vision Transformers or other emerging architectures remains an open direction.

In summary, REAL provides a principled, modular solution to the core weaknesses of ACL: it enriches the backbone’s representation through a dual‑stream pretraining and distillation pipeline, and it leverages multi‑layer features via a fusion buffer to feed the analytic classifier. The resulting system achieves state‑of‑the‑art performance on major EFCIL benchmarks while preserving the privacy‑preserving, memory‑efficient nature of exemplar‑free continual learning.