用可验证奖励强化学习,让大模型精准生成目标分子。
Adopting Reinforcement Learning with Verifiable Rewards for Molecular Generation

- 将分子设计转为条件序列预测,用可验证奖励直接指导生成
- 在多个基准上成功率更高,生成效率优于现有方法
- 适合药物分子设计、属性优化等需要精准控制的场景
利用大语言模型(LLMs)进行分子生成在化学与药物设计中展现出巨大潜力。当前方法主要依赖有限数据的监督训练或微调,难以捕捉复杂的分子设计目标。尽管部分方法尝试引导生成,但缺乏直接优化机制,难以对齐目标属性。为此,我们提出 extbf{LLMol},一种基于可验证奖励的强化学习框架,实现目标分子的直接优化。核心思想是将分子设计建模为条件序列预测任务,以可验证奖励作为显式监督信号,驱动生成向目标属性逼近。LLMol 采用两阶段训练:第一阶段对大模型进行监督微调,学习化学语法规则和分子分布;第二阶段引入基于可验证奖励的强化学习(RLVR),直接整合属性奖励信号以指导生成。为应对离散序列优化中的高方差与不稳定性,采用组相对策略优化(GRPO),平滑奖励信号并提升训练鲁棒性。该框架有效处理单属性靶向(如惩罚后 logP、QED)和结构约束优化等任务。实验表明,LLMol 在多个分子生成基准上均显著优于现有方法,成功率更高,效率更优。
原文摘要 · Abstract (English)
Leveraging large language models (LLMs) for molecular generation has shown remarkable potential in chemical and drug design. Current methods primarily rely on supervised training or fine-tuning with limited datasets, which are insufficient to capture complex molecular design objectives. While some approaches attempt to guide generation toward specific goals, they often lack direct optimization mechanisms, making it difficult to align generated molecules with desired properties. To tackle these challenges, we propose \textbf{LLMol}, a principled reinforcement learning framework that directly incorporates verifiable rewards for targeted molecule generation. The key insight is to formulate molecular design as a goal-conditioned sequence prediction task, where verifiable rewards serve as explicit supervision to drive generation toward desired objectives. LLMol follows a two-stage training paradigm combining supervised learning and reinforcement learning. In the first stage, large language models are supervised fine-tuned to capture chemical syntax and molecular distributions. In the second stage, we introduce Reinforcement Learning with Verifiable Rewards (RLVR), which directly integrates property-based reward signals to guide molecular generation toward task-specific objectives. To address the high variance and instability common in discrete sequence optimization, we adopt Group Relative Policy Optimization (GRPO), a stable on-policy algorithm that smooths reward signals and improves training robustness. This framework enables LLMol to effectively handle a range of molecular design tasks, including single-property targeting (e.g., penalized logP, QED) and structure-constrained optimization. Experimental results demonstrate that LLMol consistently outperforms existing methods, achieving higher success rates and improved efficiency across diverse molecular benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。