Multi-Task Lifelong Reinforcement Learning for Wireless Sensor Networks

Enhancing the sustainability and efficiency of wireless sensor networks (WSN) in dynamic and unpredictable environments requires adaptive communication and energy harvesting strategies. We propose a novel adaptive control strategy for WSNs that optimizes data transmission and EH to minimize overall energy consumption while ensuring queue stability and energy storing constraints under dynamic environmental conditions. The notion of adaptability therein is achieved by transferring the known environment-specific knowledge to new conditions resorting to the lifelong reinforcement learning concepts. We evaluate our proposed method against two baseline frameworks: Lyapunov-based optimization, and policy-gradient reinforcement learning (RL). Simulation results demonstrate that our approach rapidly adapts to changing environmental conditions by leveraging transferable knowledge, achieving near-optimal performance approximately $30%$ faster than the RL method and $60%$ faster than the Lyapunov-based approach. The implementation is available at our GitHub repository for reproducibility purposes [1].

💡 Research Summary

The paper addresses the challenge of jointly optimizing data transmission and energy harvesting (EH) in wireless sensor networks (WSNs) that operate under dynamically changing, non‑stationary environmental conditions. Traditional approaches—Lyapunov‑drift based optimization and conventional policy‑gradient reinforcement learning (RL)—either require re‑optimization whenever the EH statistics change or suffer from slow convergence because they cannot reuse knowledge from previous operating regimes. To overcome these limitations, the authors propose a Multi‑Task Lifelong Reinforcement Learning (MT‑L2RL) framework that treats each stationary interval of the environment (characterized by a specific energy conversion efficiency λ and EH channel scale ζ̃) as a separate “task” while keeping the state and action spaces identical across tasks.

System Model
The network consists of a primary subsystem (TX₀‑RX₀) powered by a stable source and capable of simultaneous wireless information and power transfer (SWIPT), and a secondary subsystem (TX₁‑RX₁) that relies on harvested energy from the primary node. Both subsystems maintain data queues qᵢ,t and the secondary node has a rechargeable battery bₜ. The dynamics are captured by:

1. Queue evolution (Eq. 1) based on Poisson arrivals and transmitted data.
2. Data rate (Eq. 2) determined by allocated transmission power pᵢ,t, channel gain hᵢ,t, and time fraction αᵢ,t.
3. Harvested power (Eq. 3) proportional to the residual power of the primary node and the EH channel gain ĥₜ, scaled by λ.
4. Battery update (Eq. 4) accounting for harvested energy and consumption for transmission.

The environment is non‑stationary: λ and ζ̃ change after fixed intervals of length T, defining a sequence of tasks j = 0,1,… . For each task the optimization objective is to minimize the long‑term average energy consumption (sum of pᵢ,t·αᵢ,t) while guaranteeing queue stability (average queue length → 0), battery capacity limits, and power constraints (Eqs. 6a‑6g).

Reinforcement‑Learning Formulation
The problem is cast as a Markov Decision Process (MDP) ⟨S, A, P, R, γ⟩ where:

• State s = (q₀, q₁, b, h₀, h₁, ĥ) includes queue lengths, battery level, and instantaneous channel gains.
• Action a = (p₀, α₀, α₁, α̃) specifies transmission power and time‑allocation fractions for data and EH.
• Transition dynamics P are governed by the queue and battery equations and stochastic channel variations.
• Reward R(s,a) = –∑₀¹ pᵢ·αᵢ – ν·max(b–B,0) – ∑₀¹ max(qᵢ–dᵢ,0) penalizes energy usage and constraint violations, with ν a penalty coefficient.

Standard RL would learn a separate policy πθⱼ for each task, but this ignores the shared structure across tasks and leads to slow adaptation.

Lifelong Learning Framework
MT‑L2RL maintains a global knowledge base (KB) of parameters θ that capture common features of all tasks. When a new task j arrives, the agent initializes its policy with θ and then fine‑tunes a task‑specific offset Δθⱼ. Experience replay is used to sample trajectories from both the current and past tasks, enabling the algorithm to retain useful behaviors while adapting to new EH statistics. The expected return for task j, Γ(θⱼ), is maximized, and the overall lifelong objective is to maximize the average return across an infinite sequence of tasks.

Experimental Evaluation
Simulations involve the two‑subsystem model with five consecutive tasks, each with distinct (λⱼ, ζ̃ⱼ) values. Baselines: (i) Lyapunov drift‑plus‑penalty optimization, (ii) conventional policy‑gradient RL. Metrics: average energy consumption, average queue length, and number of episodes to converge (defined as reaching within 1 % of the best‑known performance).

Results show that MT‑L2RL converges 60 % faster than the Lyapunov method and 30 % faster than the RL baseline. Energy consumption is reduced by 5‑10 % relative to both baselines, and queue stability is consistently maintained. The speedup is attributed to effective knowledge transfer: the policy learned in earlier tasks provides a good initialization for subsequent tasks, reducing exploration overhead.

Contributions and Limitations

1. Formalization of non‑stationary EH environments as a sequence of MDP tasks with shared state/action spaces, enabling lifelong learning.
2. Design of a meta‑learning based MT‑L2RL algorithm that leverages past experience for rapid adaptation.
3. Empirical demonstration of superior convergence speed and energy efficiency compared to well‑established baselines.

Limitations include the focus on a simplified two‑node topology, omission of multi‑hop routing, and reliance on perfect channel state information. Future work is suggested to extend the framework to larger networks, incorporate distributed learning, and validate on real hardware.

Overall, the paper presents a compelling approach to making WSNs more resilient and energy‑aware in the face of unpredictable environmental dynamics, by marrying multi‑task reinforcement learning with lifelong knowledge transfer.