Reinforcement Learning for Variational Quantum Circuits Design

Variational Quantum Algorithms have emerged as promising tools for solving optimization problems on quantum computers. These algorithms leverage a parametric quantum circuit called ansatz, where its parameters are adjusted by a classical optimizer with the goal of optimizing a certain cost function. However, a significant challenge lies in designing effective circuits for addressing specific problems. In this study, we leverage the powerful and flexible Reinforcement Learning paradigm to train an agent capable of autonomously generating quantum circuits that can be used as ansatzes in variational algorithms to solve optimization problems. The agent is trained on diverse problem instances, including Maximum Cut, Maximum Clique and Minimum Vertex Cover, built from different graph topologies and sizes. Our analysis of the circuits generated by the agent and the corresponding solutions shows that the proposed method is able to generate effective ansatzes. While our goal is not to propose any new specific ansatz, we observe how the agent has discovered a novel family of ansatzes effective for Maximum Cut problems, which we call $R_{yz}$-connected. We study the characteristics of one of these ansatzes by comparing it against state-of-the-art quantum algorithms across instances of varying graph topologies, sizes, and problem types. Our results indicate that the $R_{yz}$-connected circuit achieves high approximation ratios for Maximum Cut problems, further validating our proposed agent. In conclusion, our study highlights the potential of Reinforcement Learning techniques in assisting researchers to design effective quantum circuits which could have applications in a wide number of tasks.

💡 Research Summary

**
The paper tackles one of the most pressing challenges in variational quantum algorithms (VQAs): the design of an effective ansatz for a given problem. While prior work has relied on problem‑specific heuristics (symmetry‑based ansätze, hardware‑efficient designs) or adaptive schemes that iteratively add or remove gates (ADAPT‑VQE, ADAPT‑QAOA), these approaches either require substantial domain knowledge or suffer from large sample complexity when the circuit space is huge.

To address this, the authors propose a reinforcement‑learning (RL) framework called RL‑VQC (Reinforcement Learning for Variational Quantum Circuits). The RL agent interacts with an environment that represents a parametric quantum circuit on n qubits. At each time step the agent selects an action from a set consisting of (i) single‑qubit rotations (R_i^a(\theta)) (a ∈ {x,y,z}) and (ii) two‑qubit double rotations (R_{ij}^{ab}(\theta)=e^{-i\theta/2,\sigma_a\otimes\sigma_b}). The episode starts from a circuit containing only a layer of Hadamard gates; the agent may add up to a predefined maximum number of gates.

The learning algorithm is Proximal Policy Optimization (PPO), which maintains a policy network (outputting a probability distribution over actions) and a value network (estimating the expected return). The state representation combines a description of the current circuit (gate list, depth) with a feature vector of the optimization problem (graph adjacency matrix, node degrees, etc.). The reward function is a weighted sum of three components: (1) improvement in the problem’s cost function after a circuit evaluation, (2) a penalty proportional to circuit depth and number of parameters, and (3) the final approximation ratio achieved at the end of the episode. This design encourages the agent to discover short, expressive circuits that give high-quality solutions with few parameters.

Training is performed on three combinatorial‑optimization problems expressed as QUBOs: Maximum Cut, Maximum Clique, and Minimum Vertex Cover. For each problem a variety of graph families (random, regular, grid, complete) and sizes (5–15 vertices) are generated, providing a diverse curriculum.

Key empirical findings

1. Maximum Cut specialization – When trained on Max‑Cut instances, the agent repeatedly discovers a regular circuit pattern it names R_yz-connected. In this family every pair of qubits is linked by an (R_{ij}^{yz}(\theta)) gate (or a linear chain of such gates, called the Linear variant). These gates can be decomposed into native hardware operations (e.g., a CNOT‑equivalent (R_{zz}) sandwiched by single‑qubit rotations), making them hardware‑friendly.

2. Performance – The Linear R_yz circuit, even with depth p = 1–3 and only 10–30 trainable parameters, attains average approximation ratios above 0.95 on large random Max‑Cut graphs, outperforming standard QAOA (p = 1), multi‑angle QAOA, and QAOA+ which either need deeper circuits or many more parameters.

3. Generalization limits – On Max‑Clique and Minimum Vertex Cover the same R_yz-connected structures perform poorly, indicating that the discovered ansatz is highly tuned to the structure of the Max‑Cut Hamiltonian.

4. Learning dynamics – Early in training the agent explores the action space randomly; once a significant reward increase is observed, it converges to repeatedly inserting R_yz gates between specific qubit pairs. This suggests that the double‑rotation gate efficiently encodes the pairwise interaction terms of the Max‑Cut cost Hamiltonian while keeping the parameter space modest, thereby mitigating barren‑plateau effects.

5. Scalability and limitations – The action space scales as O(n²), causing training time to explode for n > 20. Moreover, the current reward formulation heavily emphasizes cost‑function reduction, making transfer to non‑binary QUBOs or continuous‑variable problems non‑trivial. All experiments are conducted in noiseless simulators; the impact of realistic hardware noise and connectivity constraints remains to be validated.

Contributions summarized

• Introduction of a PPO‑based RL agent that autonomously constructs variational quantum circuits for combinatorial optimization.
• Demonstration that the agent can achieve high approximation ratios on Max‑Cut without hand‑crafted heuristics.
• Discovery and systematic analysis of a novel ansatz family (R_yz-connected), which generalizes well across different Max‑Cut graph topologies and can be compiled efficiently on near‑term devices.

Outlook

The work opens a promising research direction where data‑driven RL replaces expert‑designed ansätze. Future extensions could involve (i) curriculum learning or meta‑RL to handle larger graphs, (ii) multi‑task training to obtain more universal ansätze, (iii) incorporation of realistic noise models into the reward, and (iv) exploration of alternative gate sets (e.g., native gates of superconducting or trapped‑ion platforms). By bridging reinforcement learning with quantum circuit synthesis, the paper provides a concrete pathway toward automated, hardware‑aware design of variational algorithms for the NISQ era.