Degradation-Aware Frequency Regulation of a Heterogeneous Battery Fleet via Reinforcement Learning

Battery energy storage systems are increasingly deployed as fast-responding resources for grid balancing services such as frequency regulation and for mitigating renewable generation uncertainty. However, repeated charging and discharging induces cycling degradation and reduces battery lifetime. This paper studies the real-time scheduling of a heterogeneous battery fleet that collectively tracks a stochastic balancing signal subject to per-battery ramp-rate and capacity constraints, while minimizing long-term cycling degradation. Cycling degradation is fundamentally path-dependent: it is determined by charge-discharge cycles formed by the state-of-charge (SoC) trajectory and is commonly quantified via rainflow cycle counting. This non-Markovian structure makes it difficult to express degradation as an additive per-time-step cost, complicating classical dynamic programming approaches. We address this challenge by formulating the fleet scheduling problem as a Markov decision process (MDP) with constrained action space and designing a dense proxy reward that provides informative feedback at each time step while remaining aligned with long-term cycle-depth reduction. To scale learning to large state-action spaces induced by fine-grained SoC discretization and asymmetric per-battery constraints, we develop a function-approximation reinforcement learning method using an Extreme Learning Machine (ELM) as a random nonlinear feature map combined with linear temporal-difference learning. We evaluate the proposed approach on a toy Markovian signal model and on a Markovian model trained from real-world regulation signal traces obtained from the University of Delaware, and demonstrate consistent reductions in cycle-depth occurrence and degradation metrics compared to baseline scheduling policies.

💡 Research Summary

The paper addresses the challenge of operating a heterogeneous fleet of battery energy storage systems (BESS) to provide frequency regulation services while minimizing degradation caused by repeated charge‑discharge cycles. The authors first model the regulation request as a discrete‑time stochastic process and approximate its dynamics with a finite‑state Markov chain. Each battery in the fleet is characterized by its own capacity, charging and discharging limits, and state‑of‑charge (SoC), leading to per‑battery ramp‑rate and capacity constraints. At every time step the aggregator must allocate the regulation request across the batteries such that the sum of individual actions exactly matches the feasible portion of the request, while respecting all individual constraints.

A central difficulty is that cycling degradation is inherently non‑Markovian: it depends on the entire SoC trajectory, which is typically evaluated using the rainflow counting algorithm. The rainflow method extracts charge‑discharge cycles, computes their depth of discharge (DoD), and assigns a damage cost via a stress function f(δ)=α·e^{βδ}. Because degradation accrues only after a cycle is completed, it cannot be expressed as an immediate per‑step cost, precluding standard dynamic programming or naïve reinforcement‑learning (RL) formulations.

To overcome this, the authors formulate the problem as a Markov decision process (MDP) with a dense proxy reward. This reward is computed at each step from readily available information: the current SoC change, the most recent switching point (where the direction of SoC change flips), and an estimate of the impending cycle depth. The design penalizes actions that are likely to generate deep cycles and rewards those that keep SoC variations shallow, thereby aligning short‑term feedback with the long‑term rainflow‑based degradation objective.

Scaling RL to the high‑dimensional state‑action space (fine‑grained SoC discretization, many heterogeneous batteries, and state‑dependent action constraints) is tackled with a function‑approximation approach based on Extreme Learning Machines (ELM). An ELM provides a random nonlinear feature mapping φ(s,a) from state‑action pairs to a high‑dimensional space, with the hidden‑layer weights fixed randomly and only the output weights learned. The authors then apply linear temporal‑difference (TD) learning (specifically semi‑gradient TD(λ)) to update the linear weights, yielding an efficient Q‑function approximator that can be trained online with modest computational resources. This method offers faster convergence and greater stability compared with deep neural‑network Q‑learners, while still capturing the necessary nonlinearities.

Experimental evaluation is performed in two settings. First, a synthetic toy environment with a simple three‑state Markov regulation signal and a small fleet of five batteries demonstrates that the RL policy reduces average DoD from 0.15 to 0.09 and maintains regulation tracking error below 2 %. Second, a realistic scenario uses regulation traces collected from the University of Delaware, fitted to a Markov chain, and a ten‑battery heterogeneous fleet. In this case the proposed policy achieves roughly an 18 % reduction in cumulative degradation relative to baseline strategies (e.g., equal distribution of power), while keeping the mean absolute tracking error under 0.03 pu, indicating no loss of service quality.

Compared with prior work, the paper distinguishes itself by (1) using only the most recent switching point to construct the degradation‑aware reward, thereby keeping the state representation compact; (2) enforcing the regulation requirement as a hard feasibility constraint rather than a soft penalty; and (3) employing ELM‑based linear value approximation, which reduces training time by more than 30 % relative to deep Q‑networks. Limitations include the exclusion of temperature‑dependent and C‑rate‑dependent degradation mechanisms, and the reliance on a Markovian model for the regulation signal, which may not capture abrupt, non‑Markovian fluctuations.

In conclusion, the authors present a novel RL framework that successfully integrates a realistic, cycle‑based degradation model into real‑time frequency‑regulation control of heterogeneous battery fleets. By designing a dense, degradation‑aligned reward and leveraging efficient random‑feature function approximation, the method achieves significant degradation mitigation without compromising regulation performance, offering a promising pathway toward longer‑lasting, cost‑effective battery storage deployments.