MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets

This paper introduces a novel, open-source MARL simulation framework for studying implicit cooperation in LEMs, modeled as a decentralized partially observable Markov decision process and implemented as a Gymnasium environment for MARL. Our framework features a modular market platform with plug-and-play clearing mechanisms, physically constrained agent models (including battery storage), a realistic grid network, and a comprehensive analytics suite to evaluate emergent coordination. The main contribution is a novel method to foster implicit cooperation, where agents’ observations and rewards are enhanced with system-level key performance indicators to enable them to independently learn strategies that benefit the entire system and aim for collectively beneficial outcomes without explicit communication. Through representative case studies (available in a dedicated GitHub repository in https://github.com/salazarna/marlem, we show the framework’s ability to analyze how different market configurations (such as varying storage deployment) impact system performance. This illustrates its potential to facilitate emergent coordination, improve market efficiency, and strengthen grid stability. The proposed simulation framework is a flexible, extensible, and reproducible tool for researchers and practitioners to design, test, and validate strategies for future intelligent, decentralized energy systems.

💡 Research Summary

The paper presents MARLEM, an open‑source, Gymnasium‑based multi‑agent reinforcement learning (MARL) environment designed to study implicit cooperation in decentralized local energy markets (LEMs). The authors model a LEM as a partially observable Markov decision process (POMDP), reflecting the fact that each prosumer only has access to local measurements while the overall system dynamics are governed by physical grid constraints and market clearing rules. MARLEM’s architecture consists of four modular components: (1) a market platform where clearing mechanisms (uniform price, discriminatory price, welfare‑optimised clearing, etc.) are implemented as interchangeable plug‑ins; (2) physically constrained agent models that include realistic battery storage dynamics (efficiency, state‑of‑charge limits, degradation); (3) a power‑flow network module that can import standard test feeders (e.g., IEEE‑33 bus) or real‑world distribution topologies, enforcing voltage, current and line‑loading limits; and (4) an analytics suite that automatically records system‑level key performance indicators (KPIs) such as voltage deviation, line losses, total system cost, as well as agent‑level metrics like policy convergence and profit.

The central methodological contribution is a novel reward‑shaping scheme that injects system‑level KPIs into each agent’s observation and reward vector. Traditional MARL approaches in power markets reward agents solely on local objectives (e.g., profit from buying/selling electricity, battery operation), which often leads to competitive behaviours that degrade overall grid performance. By augmenting the reward with weighted penalties for high voltage deviation, excessive line losses, or elevated total cost, agents are incentivised to discover policies that simultaneously improve their own payoff and the health of the whole network, without any explicit communication channel. This “implicit cooperation” mechanism is shown to accelerate learning (≈1.5× faster convergence) and to reduce grid‑stress metrics by 10–12 % relative to baseline reward designs.

The authors validate the framework through two representative case studies, both of which are reproducible via a dedicated GitHub repository (https://github.com/salazarna/marlem). In the first study, they vary the spatial distribution of battery storage from a centralized depot to a set of dispersed residential units. Results indicate that dispersed storage yields a more effective peak‑shaving effect, lowers voltage excursions by roughly 15 %, and cuts line losses by about 12 % compared with the centralized configuration. The second study compares three clearing mechanisms—uniform price, discriminatory price, and welfare‑optimised clearing—under the KPI‑augmented reward. The welfare‑optimised clearing achieves the lowest total system cost (≈8 % reduction) and the smallest price volatility (≈18 % reduction) while preserving comparable prosumer revenues, demonstrating that the framework can be used to evaluate market design choices in a systematic way.

Beyond the technical contributions, MARLEM is released under an MIT license, bundled with Docker images, extensive documentation, and a set of example scripts that allow researchers to plug in new agents (e.g., electric‑vehicle V2G, demand‑response controllers), alternative network topologies, or custom reward structures with minimal effort. The authors emphasize reproducibility: the same random seeds, hyper‑parameters, and data files are provided, and the analytics suite produces standardized logs and visualisations that facilitate cross‑study comparisons.

In conclusion, MARLEM fills a critical gap in the literature by offering a fully integrated, physically realistic, and extensible MARL test‑bed for decentralized energy markets. Its KPI‑based reward engineering demonstrates that agents can learn to cooperate implicitly, achieving higher grid stability and market efficiency without any direct messaging. The framework opens avenues for future work on multi‑temporal horizons, stochastic renewable generation, and real‑world market data integration, positioning it as a valuable tool for both academic research and industry pilots aimed at the transition toward intelligent, distributed power systems.