Agentic reinforcement learning empowers next-generation chemical language models for molecular design and synthesis

Language models are revolutionizing the biochemistry domain, assisting scientists in drug design and chemical synthesis with high efficiency. Yet current approaches struggle between small language models prone to hallucination and limited knowledge retention, and large cloud-based language models plagued by privacy risks and high inference costs. To bridge this gap, we introduce ChemCRAFT, a novel framework leveraging agentic reinforcement learning to decouple chemical reasoning from knowledge storage. Instead of forcing the model to memorize vast chemical data, our approach empowers the language model to interact with a sandbox for precise information retrieval. This externalization of knowledge allows a locally deployable small model to achieve superior performance with minimal inference costs. To enable small language models for agent-calling ability, we build an agentic trajectory construction pipeline and a comprehensive chemical-agent sandbox. Based on sandbox interactions, we constructed ChemToolDataset, the first large-scale chemical tool trajectory dataset. Simultaneously, we propose SMILES-GRPO to build a dense chemical reward function, promoting the model’s ability to call chemical agents. Evaluations across diverse aspects of drug design show that ChemCRAFT outperforms current cloud-based LLMs in molecular structure analysis, molecular optimization, and synthesis pathway prediction, demonstrating that scientific reasoning is not solely an emergent ability of model scale, but a learnable policy of tool orchestration. This work establishes a cost-effective and privacy-preserving paradigm for AI-aided chemistry, opening new avenues for accelerating molecular discovery with locally deployable agents. Code available at https://github.com/HowardLi1984/ChemCraft.

💡 Research Summary

The paper introduces ChemCRAFT, a novel framework that equips relatively small language models (7–14 B parameters) with the ability to call external chemical computation tools through a structured agentic reinforcement‑learning pipeline. The authors argue that current approaches either rely on compact models that suffer from hallucination and limited knowledge retention, or on massive cloud‑based LLMs that incur prohibitive inference costs and raise privacy concerns when proprietary molecular structures are transmitted to external services. ChemCRAFT tackles this “cost‑performance‑privacy trilemma” by explicitly decoupling chemical reasoning from knowledge storage: the language model focuses on high‑level hypothesis generation and planning, while precise calculations (e.g., SMILES parsing, QED/LogP evaluation, reaction template retrieval) are delegated to a sandbox of micro‑service tools called the Chemical Agent Sandbox.

To train such an agentic model, the authors first construct a large‑scale dataset of tool‑use trajectories, ChemToolDataset, comprising 2.9 M molecules, 1.2 M reactions, and 0.6 M property records. Unlike prior work that extracts reasoning traces from closed‑source LLMs, ChemToolDataset is built by executing a “hypothesis‑action‑observation” loop in the sandbox, recording both the textual reasoning and the concrete tool outputs. A “reflective refinement” step rewrites raw API logs into fluent, expert‑level narratives, ensuring that the model learns to interpret tool results, validate hypotheses, and adjust its plan in a human‑like manner.

Training proceeds in two stages. First, a cold‑start supervised fine‑tuning (SFT) phase teaches the model the basic syntax of chemistry and the pattern “think → call tool → observe”. Second, a reinforcement‑learning (RL) phase employs Group‑Relative Policy Optimization (GRPO) together with a bespoke dense reward function named SMILES‑GRPO. This reward aggregates multiple chemically meaningful signals: exact SMILES matching, scaffold similarity, functional‑group alignment, reaction‑template concordance, and quantitative property improvements (ΔLogP, ΔQED). By optimizing against these granular metrics rather than simple text overlap, the model learns to orchestrate tool calls that maximize scientific validity.

Evaluation is carried out on ChemCoT‑Bench, a comprehensive benchmark that decomposes drug‑discovery workflows into nine major tasks and twenty‑two subtasks, ranging from basic molecule understanding and editing to advanced molecular optimization and retrosynthesis planning. ChemCRAFT consistently outperforms both open‑source baselines (e.g., Gemini‑2.5‑Pro, DeepSeek‑V3, Qwen‑2.5‑32B) and commercial APIs (Claude‑3.7‑sonnet, GPT‑4). Notably, on functional‑group detection the model achieves a mean absolute error of 0.03, on ring‑system detection it reaches 100 % accuracy, and on molecule‑editing operations it attains roughly 95 % correctness. In property‑guided optimization and reaction‑prediction tasks, the 14 B‑parameter ChemCRAFT matches or exceeds the performance of much larger proprietary models, demonstrating that effective tool orchestration can compensate for limited model size.

The authors discuss several limitations. The current sandbox includes only a core set of cheminformatics utilities; extending it to handle metal complexes, polymers, or catalyst design would require additional services. The multi‑objective reward design is intricate and demands careful hyper‑parameter tuning to maintain policy stability during RL. Nonetheless, the work showcases a viable path toward locally deployable, privacy‑preserving AI chemists that combine the reasoning flexibility of language models with the precision of specialized computational tools. Future directions include expanding the toolbox, integrating multimodal inputs (e.g., 3D structures, spectra), and developing interactive human‑AI collaboration interfaces.