An Attention-based Feature Memory Design for Energy-Efficient Continual Learning

Tabular data streams are increasingly prevalent in real-time decision-making across healthcare, finance, and the Internet of Things, often generated and processed on resource-constrained edge and mobile devices. Continual learning (CL) enables models to learn sequentially from such streams while retaining previously acquired knowledge. While recent CL advances have made significant progress in mitigating catastrophic forgetting, the energy and memory efficiency of CL for tabular data streams remains largely unexplored. To address this gap, we propose AttenMLP, which integrates attention-based feature replay with context retrieval and sliding buffer updates within a minibatch training framework for streaming tabular learning. We evaluate AttenMLP against state-of-the-art (SOTA) tabular models on real-world concept drift benchmarks with temporal distribution shifts. Experimental results show that AttenMLP achieves accuracy comparable to strong baselines without replay, while substantially reducing energy consumption through tunable design choices. In particular, with the proposed attention-based feature memory design, AttenMLP costs a 0.062 decrease in final accuracy under the incremental concept drift dataset, while reducing energy usage up to 33.3% compared to TabPFNv2. Under the abrupt concept drift dataset, AttenMLP reduces 1.47% energy consumption compared to TabR, at the cost of a 0.038 decrease in final accuracy. Although ranking third in global efficiency, AttenMLP demonstrates energy-accuracy trade-offs across both abrupt and incremental concept drift scenarios compared to SOTA tabular models.

💡 Research Summary

The paper introduces AttenMLP, a novel energy‑efficient continual learning (EECL) framework designed for streaming tabular data on resource‑constrained edge and mobile devices. While most continual learning research has focused on vision and language, the authors identify a gap in energy‑aware methods for tabular streams, where replay‑based approaches typically require unbounded memory to store raw samples, leading to high power consumption.

Problem setting – The model receives a sequence of tasks T₁,…,Tₜ, each providing a batch of feature vectors Xₜ and labels Yₜ drawn from a distribution Dₜ that may drift over time (Dₜ ≠ Dₜ₊₁). The goal is to update the predictor online, retain knowledge from previous tasks, and do so with a strict energy budget, without ever accessing raw past data.

Core architecture – AttenMLP builds a fixed‑size feature memory of the most recent W latent vectors hₜ,ⱼ ∈ ℝ^{d_h}. Instead of storing raw inputs, the system stores only these compact embeddings. At each step a scaled‑dot‑product attention is applied: the current input xₜ is linearly projected to a query qₜ, the memory matrix Hₜ is projected to keys Kₜ, and attention scores are computed as sₜ = qₜ·Kₜᵀ. After scaling by 1/√{d_h} and a softmax, the context vector cₜ = αₜᵀ Kₜ is obtained as a weighted sum of past keys. This context is concatenated with the raw input, forming \tilde{x}_t =