Tackling GNARLy Problems: Graph Neural Algorithmic Reasoning Reimagined through Reinforcement Learning

Neural Algorithmic Reasoning (NAR) is a paradigm that trains neural networks to execute classic algorithms by supervised learning. Despite its successes, important limitations remain: inability to construct valid solutions without post-processing and to reason about multiple correct ones, poor performance on combinatorial NP-hard problems, and inapplicability to problems for which strong algorithms are not yet known. To address these limitations, we reframe the problem of learning algorithm trajectories as a Markov Decision Process, which imposes structure on the solution construction procedure and unlocks the powerful tools of imitation and reinforcement learning (RL). We propose the GNARL framework, encompassing the methodology to translate problem formulations from NAR to RL and a learning architecture suitable for a wide range of graph-based problems. We achieve very high graph accuracy results on several CLRS-30 problems, performance matching or exceeding much narrower NAR approaches for NP-hard problems and, remarkably, applicability even when lacking an expert algorithm.

💡 Research Summary

The paper tackles fundamental shortcomings of Neural Algorithmic Reasoning (NAR) by recasting algorithm execution as a Markov Decision Process (MDP) and training a policy with reinforcement learning (RL) and imitation learning (IL). Traditional NAR relies on supervised learning of intermediate “hints” generated by a known algorithm; this approach suffers from three major drawbacks: (1) solutions often require post‑processing to become valid, (2) the framework cannot naturally handle problems with multiple correct outputs, and (3) it is limited to problems for which a strong expert algorithm exists, making it difficult to apply to NP‑hard combinatorial optimisation tasks.

The authors propose the Graph Neural Algorithmic Reasoning with Reinforcement Learning (GNARL) framework. They define a generic graph‑algorithm MDP, MA, where the state consists of the original input features together with algorithmic state features (including a “phase” indicator and previously selected node/edge information). Actions are selections of graph elements (nodes, edges, or triangles) and the transition function updates the internal state according to the underlying algorithmic logic. Rewards are shaped as the difference in a problem‑specific objective J between successive states, which, by the reward‑shaping theorem, yields the same optimal policy as a terminal‑only reward. The horizon is set to the worst‑case number of steps, independent of the number of hints used in NAR.

GNARL’s neural architecture builds on the classic encode‑process‑decode paradigm of NAR but augments it with phase‑aware node and edge embeddings. A message‑passing neural network (MPNN) propagates information at each step, while an actor‑critic model (implemented with Proximal Policy Optimisation, PPO) learns the policy. When an expert algorithm is available, Behavioural Cloning (BC) is used to pre‑train the actor on state‑action pairs; PPO then fine‑tunes the policy. When no expert exists, the system is trained purely with PPO using the shaped reward.

Empirical evaluation covers the CLRS‑30 benchmark (30 classic algorithms) and several NP‑hard problems such as Maximum Independent Set, Travelling Salesperson, and Maximum Cut. GNARL achieves graph‑accuracy (the proportion of graphs for which all node labels are correct) above 90 % on most CLRS‑30 tasks, surpassing prior NAR models that often fall below 50 % on out‑of‑distribution sizes. For NP‑hard problems, GNARL matches or exceeds specialised NAR‑based approaches while producing valid solutions without any post‑processing. Notably, the framework also succeeds on a novel problem—Robust Graph Construction—where no expert algorithm is known; the RL‑only policy learns to optimise the target objective effectively.

The paper discusses limitations: (i) defining the MDP (phases, transition rules) still requires problem‑specific engineering, (ii) scalability to very large graphs (thousands of nodes) is limited by sample efficiency, and (iii) reward design can be sensitive for multi‑objective settings. Future work is suggested on automated MDP extraction, learned reward functions via inverse RL, distributed training for large‑scale graphs, and meta‑RL for algorithm discovery.

In summary, GNARL demonstrates that viewing algorithmic reasoning as sequential decision‑making unifies polynomial‑time and combinatorial‑optimisation problems under a single learning framework. By leveraging RL’s ability to enforce valid construction and handle multiple solutions, GNARL overcomes the core drawbacks of NAR, opening a path toward neural models that can both emulate known algorithms and discover new ones for graph‑structured tasks.