Hybrid Action Reinforcement Learning for Quantum Architecture Search

Reinforcement learning-based Quantum Architecture Search (QAS) offers a promising avenue for automating the design of variational quantum circuits, but existing methods typically decouple discrete structure search from continuous parameter optimization, resulting in inefficient or brittle solutions. We propose HyRLQAS (Hybrid-Action Reinforcement Learning for Quantum Architecture Search), a unified reinforcement learning framework that jointly learns gate placement and parameter initialization within a hybrid discrete-continuous action space, while enabling dynamic refinement of previously placed gates. Trained in a variational quantum eigensolver setting, the agent constructs circuits that directly optimize molecular ground-state energies. Across multiple molecular benchmarks, HyRLQAS demonstrates strong and competitive performance against state-of-the-art QAS methods, achieving lower energy errors with fewer gates. Notably, HyRLQAS reaches chemical-accuracy-level convergence down to 1e-8 energy error after classical optimization, and policy-guided initialization reduces the iteration count of downstream classical optimizers. These results demonstrate that hybrid-action reinforcement learning provides a principled and effective mechanism for coupling circuit topology design with optimization-aware parameterization.

💡 Research Summary

The paper introduces HyRLQAS, a novel hybrid‑action reinforcement‑learning framework that simultaneously searches for quantum circuit topology and initializes variational parameters for variational quantum eigensolver (VQE) tasks. Traditional quantum architecture search (QAS) methods treat gate placement (discrete decisions) and parameter optimization (continuous decisions) as separate stages, which leads to inefficient exploration, poor utilization of optimization experience, and sensitivity to parameter initialization. HyRLQAS resolves these issues by defining a unified action space composed of a discrete component z (gate type and location) and a continuous component x (initial rotation angle). The agent observes a state sₜ encoded as a tensor representation of the partially built circuit and selects (zₜ, xₜ) at each step.

The policy network consists of a discrete head that chooses among 3N single‑qubit rotations (RX, RY, RZ) and N(N‑1)/2 CNOTs (for an N‑qubit device), with illegal actions masked out dynamically, and a continuous head that outputs the mean µₜ and standard deviation σₜ of a Gaussian distribution from which xₜ is sampled. When a new parameterized gate is added, a refinement step adds a Gaussian‑distributed increment to all existing rotation parameters, allowing the policy to continuously adapt previously set angles as the circuit grows.

Reward shaping is based on the VQE energy Eₜ of the current circuit after a brief classical optimization of all parameters. A curriculum‑driven threshold ξ is gradually lowered; reaching it yields a +5 terminal reward, while failing to meet it after the maximum episode length yields –5. Intermediate steps receive a normalized improvement reward proportional to (Eₜ₋₁ − Eₜ)/(Eₜ₋₁ − E_min). Episodes terminate either when the energy falls below ξ or when a stochastic halting condition reaches a predefined maximum length ℓ.

Training employs REINFORCE with a baseline, updating both discrete and continuous policy parameters. Crucially, the continuous policy learns a distribution over initial angles that captures knowledge from previous optimization runs, so that each new circuit starts from a “policy‑guided” initialization rather than a random guess. This reduces the number of iterations required by downstream classical optimizers (e.g., COBYLA, Adam) and improves final convergence.

The authors analyze the effect of policy‑guided initialization using the Quantum Neural Tangent Kernel (QNTK) and its dynamic variant (dQNTK). They show that the learned initialization improves kernel conditioning, leading to better gradient propagation and mitigating barren‑plateau phenomena, which explains the observed stability and faster convergence.

Empirical evaluation is performed on three standard molecular benchmarks (LiH, BeH₂, H₂O) in the STO‑3G basis with symmetry‑reduced qubit encodings (4–6 qubits). HyRLQAS is compared against state‑of‑the‑art QAS methods: CR‑LQAS, TensorRL‑QAS, evolutionary QAS, and fixed hardware‑efficient ansätze. Results demonstrate:

1. Higher accuracy with fewer gates – For a comparable gate budget, HyRLQAS achieves 30–50 % lower energy error. In the H₂O case it reaches an absolute error below 1 × 10⁻⁸ Hartree, surpassing chemical‑accuracy (≈1 kcal/mol).
2. Reduced optimizer iterations – The policy‑guided initialization cuts the average number of classical optimizer steps by roughly 40 % across all molecules, with the most pronounced gain on the more complex BeH₂.
3. Benefit of refinement – Ablation without the refinement step (parameter increments) leads to slower convergence and up to a three‑fold increase in final energy error, highlighting the importance of dynamically updating previously set angles.

The paper also discusses limitations: experiments are limited to small‑scale circuits; the policy network currently uses multilayer perceptrons rather than more expressive graph neural networks; and hardware noise is not explicitly modeled. Future work includes scaling to larger qubit counts, integrating more sophisticated encoders, and testing on real NISQ devices.

In summary, HyRLQAS provides a principled, unified approach to quantum architecture search by coupling discrete gate placement with continuous parameter initialization within a single reinforcement‑learning policy. The hybrid‑action formulation, curriculum‑driven reward, and policy‑guided initialization together yield circuits that are both more compact and more amenable to downstream classical optimization, establishing a significant advance over existing QAS techniques.