Modular Multi-Task Learning for Chemical Reaction Prediction

Adapting large language models (LLMs) trained on broad organic chemistry to smaller, domain-specific reaction datasets is a key challenge in chemical and pharmaceutical R&D. Effective specialisation requires learning new reaction knowledge while preserving general chemical understanding across related tasks. Here, we evaluate Low-Rank Adaptation (LoRA) as a parameter-efficient alternative to full fine-tuning for organic reaction prediction on limited, complex datasets. Using USPTO reaction classes and challenging C-H functionalisation reactions, we benchmark forward reaction prediction, retrosynthesis and reagent prediction. LoRA achieves accuracy comparable to full fine-tuning while effectively mitigating catastrophic forgetting and better preserving multi-task performance. Both fine-tuning approaches generalise beyond training distributions, producing plausible alternative solvent predictions. Notably, C-H functionalisation fine-tuning reveals that LoRA and full fine-tuning encode subtly different reactivity patterns, suggesting more effective reaction-specific adaptation with LoRA. As LLMs continue to scale, our results highlight the practicality of modular, parameter-efficient fine-tuning strategies for their flexible deployment for chemistry applications.

💡 Research Summary

This paper investigates Low‑Rank Adaptation (LoRA), a parameter‑efficient fine‑tuning technique, as an alternative to conventional full fine‑tuning for adapting large language models (LLMs) to specialized chemical reaction datasets. The authors first pre‑train or fine‑tune T5‑style models (ByT5‑small, ByT5‑base, and nach0) on a broad organic chemistry corpus (USPTO 1K TPL). They then apply either full fine‑tuning or LoRA to adapt these base models to ten randomly selected reaction classes from the same dataset, evaluating three inter‑related tasks: forward reaction prediction, retrosynthesis, and reagent prediction.

LoRA works by freezing the original weight matrices and inserting two low‑rank matrices (A and B) whose product ΔW = B·A provides a lightweight task‑specific adjustment. With rank r = 4–16, the number of trainable parameters drops to less than 1 % of the full model, dramatically reducing computational cost and memory usage.

Experimental results show that both full fine‑tuning and LoRA improve over a direct evaluation of the general model across all tasks. Forward reaction prediction already exceeds 90 % accuracy for all configurations, reflecting the strong one‑to‑one mapping in the USPTO data. For retrosynthesis and reagent prediction, LoRA achieves marginally higher average accuracies (0.3–2 % improvement) than full fine‑tuning, and larger rank settings (r = 16) yield the best performance. Statistical tests (Wilcoxon, p < 0.05) confirm that the improvements are significant across the ten classes.

To assess chemical generalisation, the authors analyse reagent embeddings produced by the models. Both approaches can suggest out‑of‑distribution reagents, but full fine‑tuning generates a larger and more diverse set, while LoRA’s suggestions cluster more tightly, indicating that LoRA preserves the base model’s knowledge while learning task‑specific nuances.

The most demanding benchmark is a metal‑catalysed C–H borylation dataset, which involves subtle regio‑selectivity governed by directing groups, steric effects, and catalyst choice. Direct evaluation of the general model fails (Acc@1 ≈ 0.7 %). Full fine‑tuning of ByT5‑small reaches 69.6 % top‑1 accuracy, whereas LoRA (r = 16, α = 32, learning rate = 0.003) pushes this to 78.3 % (ByT5‑base reaches 79.7 %). Error analysis reveals that the full‑fine‑tuned model tends to predict the electronically most activated site, while LoRA more often selects the site dictated by dataset‑specific directing effects, suggesting that LoRA captures finer reaction‑specific patterns.

Overall, LoRA offers four key advantages: (1) extreme parameter efficiency (≈0.1–0.2 % of total weights), (2) mitigation of catastrophic forgetting by keeping the bulk of the pretrained knowledge intact, (3) better preservation of multi‑task performance, and (4) enhanced ability to learn subtle, reaction‑specific reactivity patterns. Its modular nature allows multiple LoRA adapters to be swapped in and out without retraining the base model, facilitating rapid adaptation to new reaction classes—a practical requirement in pharmaceutical and chemical R&D.

The authors conclude that modular, parameter‑efficient fine‑tuning strategies like LoRA are poised to become essential tools for deploying ever‑larger LLMs in chemistry, enabling flexible, knowledge‑preserving specialization across diverse reaction domains. Future work will extend the approach to newer GPT‑style models, larger and more diverse reaction datasets, and multimodal inputs (text, graphs, images) to further broaden the impact of modular adaptation in chemical AI.