Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation

We introduce $\textbf{Hierarchical Taylor Series-based Continual Learning (HTCL)}$, a framework that couples fast local adaptation with conservative, second-order global consolidation to address the high variance introduced by random task ordering. To address task-order effects, HTCL identifies the best intra-group task sequence and integrates the resulting local updates through a Hessian-regularized Taylor expansion, yielding a consolidation step with theoretical guarantees. The approach naturally extends to an $L$-level hierarchy, enabling multiscale knowledge integration in a manner not supported by conventional single-level CL systems. Across a wide range of datasets and replay and regularization baselines, HTCL acts as a model-agnostic consolidation layer that consistently enhances performance, yielding mean accuracy gains of $7%$ to $25%$ while reducing the standard deviation of final accuracy by up to $68%$ across random task permutations.

💡 Research Summary

**
Continual learning (CL) aims to train a single neural network on a stream of tasks without catastrophically forgetting previously learned knowledge. While replay, regularization, and architecture‑based methods have made substantial progress in mitigating forgetting, they remain highly sensitive to the order in which tasks are presented. In real‑world deployments, task order is often uncontrollable, and random permutations can cause performance variance as high as 20–30 % across runs. This “task‑order sensitivity” is a critical barrier to reliable deployment of CL systems.

The paper introduces Hierarchical Taylor Series‑based Continual Learning (HTCL), a novel framework that explicitly tackles task‑order sensitivity and forgetting by (1) partitioning the task stream into small groups, (2) exhaustively evaluating all intra‑group permutations to select the best ordering, and (3) consolidating the resulting locally optimal models into a global model using a Hessian‑regularized second‑order Taylor expansion.

Group‑wise order optimization
Given t tasks, HTCL splits them into m disjoint groups of size k (with t ≈ m·k). Instead of searching the factorial space t! of all possible permutations, HTCL only needs to evaluate k! permutations per group, reducing the combinatorial burden to m·k!. The authors prove (Theorem A.3) that selecting the best intra‑group ordering yields an expected loss that is at least as good as the average loss over a uniformly random ordering, providing a theoretical guarantee for the grouping strategy.

Local learning
Within each group, any existing CL algorithm (e.g., Experience Replay, DER, SER, EWC, SI) can be employed as a “local learner”. The local model wₗ is trained aggressively to exploit the optimal intra‑group order, achieving high plasticity for the current tasks while being invariant to their internal sequence.

Second‑order hierarchical consolidation
After a group is solved, its locally optimal parameters must be merged into a higher‑level “hierarchical” model w₁, w₂, …, w_L. Simple averaging or direct replacement would treat all parameter directions equally, ignoring that some directions are highly sensitive to previously learned tasks. HTCL therefore constructs a quadratic surrogate of the cumulative loss J(w) around the previous hierarchical weights w₁⁻: \