Market-Based Model in CR-WSN: A Q-Probabilistic Multi-agent Learning Approach

The ever-increasingly urban populations and their material demands have brought unprecedented burdens to cities. Smart cities leverage emerging technologies like the Internet of Things (IoT), Cognitive Radio Wireless Sensor Network (CR-WSN) to provide better QoE and QoS for all citizens. However, resource scarcity is an important challenge in CR-WSN. Generally, this problem is handled by auction theory or game theory. To make CR-WSN nodes intelligent and more autonomous in resource allocation, we propose a multi-agent reinforcement learning (MARL) algorithm to learn the optimal resource allocation strategy in the oligopoly market model. Firstly, we model a multi-agent scenario, in which the primary users (PUs) is the sellers and the secondary users (SUs) is the buyers. Then, we propose the Q-probabilistic multiagent learning (QPML) and apply it to allocate resources in the market. In the multi-agent interactive learning process, the PUs and SUs learn strategies to maximize their benefits and improve spectrum utilization. Experimental results show the efficiency of our QPML approach, which can also converge quickly.

💡 Research Summary

**
The paper addresses the pressing challenge of spectrum resource allocation in Cognitive Radio Internet of Things (CR‑IoT) or Cognitive Radio Wireless Sensor Networks (CR‑WSN), which are essential for the operation of smart cities. Traditional approaches based on game theory or auction mechanisms are largely centralized and lack the adaptability required for real‑time, autonomous decision making. To overcome these limitations, the authors model the interaction between two primary users (PUs) acting as sellers and multiple secondary users (SUs) as buyers using a Bertrand oligopolistic market framework. Within this market, PUs set prices for their sub‑channels, while SUs submit bids based on their demand, creating a dynamic feedback loop.

The core contribution is the Q‑Probabilistic Multi‑Agent Learning (QPML) algorithm, a multi‑agent reinforcement learning (MARL) method that extends classic Q‑learning with a probabilistic action‑selection mechanism. By treating the problem as a stateless distributed multi‑agent dynamic resource allocation problem (DMDRAP), each agent updates its policy solely based on immediate reward (revenue) without relying on historical states. The probabilistic policy enables a balance between exploration and exploitation, allowing both PUs and SUs to converge toward strategies that maximize individual profit while improving overall spectrum utilization.

Experimental evaluation compares QPML against conventional game‑theoretic, auction‑based, and single‑agent Q‑learning schemes in simulations with two PUs and 5–10 SUs. Results show that QPML converges significantly faster (within 200–300 episodes), achieves higher average revenues for both sellers and buyers, and improves total spectrum utilization by roughly 10–15 %. These findings demonstrate the effectiveness of the proposed market‑based MARL approach for distributed, autonomous spectrum allocation.

Nevertheless, the study has notable limitations. The simulations assume ideal channel conditions, ignoring noise, interference, and mobility, which may affect real‑world performance. The use of only two PUs restricts the generality of the Bertrand model for larger, more complex markets. Moreover, the paper lacks a rigorous theoretical proof of convergence for the probabilistic Q‑learning variant and provides limited sensitivity analysis of key hyper‑parameters.

In summary, the work introduces a novel market‑driven MARL framework for CR‑IoT/CR‑WSN resource allocation, showing promising empirical advantages over existing methods. Future research should extend the model to realistic wireless environments, scale to larger numbers of agents, and deepen the theoretical understanding of the Q‑probabilistic learning dynamics.