One Hop Closer to Chemistry Superintelligence: Leveraging Knowledge Graphs in Reinforcement Learning Models

Abstract

Large language models (LLMs) excel with single-hop fact queries but struggle with multi-hop reasoning tasks in specialized scientific domains such as chemistry. We thus investigate whether grounding a model in structured relational knowledge from the onset can improve how it learns to reason across multiple steps. To do this, we extend the bottom-up domain-specific framework of Dedhia et al. to the chemical domain. Using the Chemical Entities of Biological Interest (ChEBI) ontology, we construct a curated knowledge graph (KG) of 53,226 nodes and 90,470 edges. Then, we enumerate all k-hop paths of length 1–3 from the filtered KG, yielding 158,036 paths. These paths are converted into multiple-choice question-answer (QA) pairs with step-by-step reasoning traces via Gemini 2.5 Flash, before being cross-validated by DeepSeek-V3. The resulting 119,006 validated 1- and 2-hop QA pairs are used to fine-tune Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct under a progressive hop-splitting paradigm, where models are trained on QA pairs derived from 1- and 2-hop paths and evaluated on unseen 3-hop chains. Supervised fine-tuning (SFT) yields consistent improvements of ~13 percentage points over baseline for both model sizes (38.8% → 52.1% for 7B; 42.8% → 57.0% for 14B) on a held-out test set of 13,499 3-hop questions. The 7B SFT model is refined via Group Relative Policy Optimization (GRPO) with KG-derived rewards combining binary correctness and path alignment signals, yielding 70.4% accuracy. Our results demonstrate that bottom-up KG-grounded curriculum learning improves generalization to longer-hop compositional reasoning tasks.