Pushing the Limits of Distillation-Based Class-Incremental Learning via Lightweight Plugins

Existing replay and distillation-based class-incremental learning (CIL) methods are effective at retaining past knowledge but are still constrained by the stability-plasticity dilemma. Since their resulting models are learned over a sequence of incremental tasks, they encode rich representations and can be regarded as pre-trained bases. Building on this view, we propose a plug-in extension paradigm termed Deployment of LoRA Components (DLC) to enhance them. For each task, we use Low-Rank Adaptation (LoRA) to inject task-specific residuals into the base model’s deep layers. During inference, representations with task-specific residuals are aggregated to produce classification predictions. To mitigate interference from non-target LoRA plugins, we introduce a lightweight weighting unit. This unit learns to assign importance scores to different LoRA-tuned representations. Like downloadable content in software, DLC serves as a plug-and-play enhancement that efficiently extends the base methods. Remarkably, on the large-scale ImageNet-100, with merely 4% of the parameters of a standard ResNet-18, our DLC model achieves a significant 8% improvement in accuracy, demonstrating exceptional efficiency. Under a fixed memory budget, methods equipped with DLC surpass state-of-the-art expansion-based methods.

💡 Research Summary

Class‑incremental learning (CIL) aims to continuously acquire new classes while preserving knowledge of previously seen ones. Existing CIL approaches fall into three main families: rehearsal‑based methods that store a small exemplar buffer, restriction‑based methods that rely on knowledge distillation to align the current model’s outputs with those of a previous “teacher” model, and expansion‑based methods that allocate separate network blocks for each task. Rehearsal methods incur memory overhead proportional to the number of stored images, while restriction‑based (distillation) methods suffer from the classic stability‑plasticity dilemma: the distillation loss that preserves old knowledge conflicts with the classification loss for the new task, limiting overall accuracy. Expansion‑based approaches avoid catastrophic forgetting by isolating parameters per task, but they dramatically increase model size—often adding a full or partial feature extractor for every new task, which is impractical for long task sequences or resource‑constrained devices.

The paper proposes a novel paradigm called Deployment of LoRA Components (DLC) that bridges the gap between the parameter‑efficiency of distillation‑based CIL and the performance gains of expansion‑based CIL. The key insight is to treat a replay‑and‑distillation trained network as a “base model” that already encodes rich representations, and then augment it with lightweight, task‑specific plugins. Each plugin is instantiated using Low‑Rank Adaptation (LoRA), a parameter‑efficient fine‑tuning technique originally designed for large language models. LoRA injects a low‑rank residual ΔW = A·B into a frozen weight matrix W, where A∈ℝ^{d×r} and B∈ℝ^{r×d} (r≪d). This adds only O(r·d) parameters per plugin, typically a few percent of the original model size.

For every incremental task t, DLC creates a dedicated set L_t of k LoRA plugins and attaches them to the deep layers of the base feature extractor ϕ. The first plugin is placed on the last convolutional layer, and subsequent plugins are positioned progressively earlier in the network. This deep‑layer placement is motivated by two considerations: (1) deeper features are more semantically meaningful and directly influence the classifier, so residual injection there yields maximal discriminative benefit; (2) shallow‑layer modifications would propagate through the network and amplify logit deviations, making the distillation loss harder to satisfy.

Training proceeds in two decoupled phases. Phase 1 follows the underlying CIL baseline: the base extractor ϕ and the classifier W are updated using the current task data D_t together with replay exemplars D_exp, under the combined loss L = L_CE + L_KD (cross‑entropy plus knowledge‑distillation). All plugins remain frozen. Phase 2 freezes ϕ and W, then trains only the task‑specific plugin set L_t on the same data, using a loss L_plg = L_CE + L_aux, where L_aux is a lightweight auxiliary term borrowed from expansion‑style methods. Because the base model does not change during plugin training, the plugins can specialize without interfering with the distillation objective, and the base model’s parameters are protected from drift.

During inference, every plugin L_t is activated, producing a set of task‑specific feature vectors. These vectors are concatenated and fed to a lightweight weighting unit placed before the final classifier. The weighting unit learns a scalar importance α_t for each plugin’s output (implemented as a simple fully‑connected layer followed by a softmax). This mechanism suppresses noisy contributions from plugins that are irrelevant to the current input, thereby mitigating feature interference across tasks.

The authors provide a theoretical analysis showing that the output deviation introduced by a LoRA plugin at layer ℓ is bounded by K_ℓ·Γ_t, where K_ℓ depends on network architecture and distillation temperature, and Γ_t captures the distribution shift induced by the replay‑distillation strategy. This bound guarantees that, under typical replay‑distillation regimes, the feature drift caused by plugins remains limited, preserving the effectiveness of the distillation loss.

Empirical evaluation is conducted on ImageNet‑100 (using ResNet‑18) and CIFAR‑100 (using ResNet‑32). DLC adds only about 4 % of the original parameters (≈0.7 M additional weights for ResNet‑18) yet yields an 8 % absolute accuracy gain on ImageNet‑100 compared to the vanilla distillation baseline. When integrated with popular CIL methods—iCaRL, WA, and BiC—the DLC‑enhanced versions (iCaRL‑DLC, WA‑DLC, BiC‑DLC) consistently outperform state‑of‑the‑art expansion‑based approaches while respecting the same memory budget for replay buffers. Notably, DLC achieves a better trade‑off between parameter efficiency and accuracy than methods that allocate a full or partial feature extractor per task (which can add ≈11.7 MB per task for ResNet‑18).

In summary, DLC introduces a plug‑and‑play extension framework that leverages low‑rank residual adapters to provide task‑specific capacity without bloating the model. The lightweight weighting unit ensures that only relevant plugin information contributes to the final prediction, preserving stability while enhancing plasticity. This approach is especially attractive for edge devices and long‑running continual learning scenarios where memory and compute resources are limited, yet high accuracy across many incremental tasks is required.