PANDA -- Patch And Distribution-Aware Augmentation for Long-Tailed Exemplar-Free Continual Learning

Exemplar-Free Continual Learning (EFCL) restricts the storage of previous task data and is highly susceptible to catastrophic forgetting. While pre-trained models (PTMs) are increasingly leveraged for EFCL, existing methods often overlook the inherent imbalance of real-world data distributions. We discovered that real-world data streams commonly exhibit dual-level imbalances, dataset-level distributions combined with extreme or reversed skews within individual tasks, creating both intra-task and inter-task disparities that hinder effective learning and generalization. To address these challenges, we propose PANDA, a Patch-and-Distribution-Aware Augmentation framework that integrates seamlessly with existing PTM-based EFCL methods. PANDA amplifies low-frequency classes by using a CLIP encoder to identify representative regions and transplanting those into frequent-class samples within each task. Furthermore, PANDA incorporates an adaptive balancing strategy that leverages prior task distributions to smooth inter-task imbalances, reducing the overall gap between average samples across tasks and enabling fairer learning with frozen PTMs. Extensive experiments and ablation studies demonstrate PANDA’s capability to work with existing PTM-based CL methods, improving accuracy and reducing catastrophic forgetting.

💡 Research Summary

This paper addresses a critical challenge in Exemplar-Free Continual Learning (EFCL): learning from non-stationary data streams that exhibit severe and complex imbalances. The authors identify that real-world data often suffers from a “Dual-Level Imbalance” (DLI), which consists of not only a global long-tailed distribution across the entire dataset but also extreme or even reversed skews within individual tasks. This dual imbalance exacerbates catastrophic forgetting and biases learning towards the most frequent classes within each task, hindering generalization.

To tackle this problem, the authors propose PANDA, a novel Patch-and-Distribution-Aware Augmentation framework designed to integrate seamlessly with existing Pre-trained Model (PTM) based EFCL methods. PANDA operates through two synergistic mechanisms. First, for intra-task balancing, it performs intelligent oversampling for tail (rare) classes. It utilizes a frozen CLIP encoder to identify the most semantically representative image patches (e.g., the head of a rare animal) within tail-class samples. These informative patches are then surgically transplanted onto the contextual background of head-class (frequent) samples, creating new, context-rich synthetic samples for the tail classes. This approach enriches the feature diversity of tail classes without simply replicating limited existing data.

Second, to mitigate inter-task distribution shifts, PANDA employs an adaptive distribution smoothing strategy. It maintains a lightweight memory of the statistical characteristics (e.g., min/max feature values) observed from prior tasks. When learning a new task, it blends these historical statistics with those of the current task using a learnable blending coefficient, beta. This beta is adjusted dynamically based on performance, promoting stability when learning is effective and allowing for adaptation when facing difficult shifts. This process smooths the transition across tasks and reduces classifier bias introduced by sequential learning with a frozen PTM backbone.

The efficacy of PANDA is rigorously evaluated on long-tailed versions of CIFAR-100 and a 100-class subset of the naturally imbalanced iNaturalist dataset. Experiments are conducted under both Single-Level Imbalance (SLI) and the more challenging Dual-Level Imbalance (DLI) settings. PANDA is integrated as a plug-in module into several state-of-the-art PTM-based EFCL methods, including prompt-based techniques like L2P, CodaPrompt, DualPrompt, and DAP. The results demonstrate consistent and significant improvements. For instance, on the imbalanced CIFAR-100-LT, integrating PANDA with CodaPrompt boosted the final average accuracy from 76.52% to 87.49% while reducing average forgetting from 7.55% to 4.61%. Similar gains were observed across all base methods and on the iNaturalist dataset. Ablation studies confirm the individual contribution of both the patch-augmentation and distribution-smoothing components.

In summary, PANDA provides a practical and effective solution to the underexplored problem of dual-level imbalance in EFCL. Its training-free, plug-and-play nature allows it to enhance a wide range of existing methods, leading to more accurate and robust continual learning systems that are better aligned with the complexities of real-world data streams.