arXiv:2607.00531cs.LGcs.AI2026-07

让模型自主决定何时模仿参考、何时自我改进,提升分子优化效率。

Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization

论文配图:Active-GRPO: Adaptive Imitation and Self-Improving Reasoning for Molecular Optimization
图 1 · 摘自论文原文
  • 模型按需切换模仿与自研策略,动态更新参考标准。
  • 在分子优化任务中,成功率从0.1665提升至0.1773。
  • 适合需要高效自进化推理的化学生成与药物设计场景。

科学推理是大语言模型的重要能力,但提升其训练鲁棒性与效率仍是关键挑战。本文聚焦基于指令的分子优化任务,发现仅用答案监督微调会丢失多步推理,而基于可验证奖励的强化学习则面临反馈稀疏问题。参考引导策略优化虽缓解二者缺陷,但性能受限于参考质量。为此,提出主动推理范式:模型在每例中自主判断是否模仿参考或强化自身发现,并持续升级参考。构建主动组相对策略优化(Active-GRPO),通过主动模仿-强化与主动引用两个机制实现。前者在参考优于模型时模仿,一旦模型生成更优分子则转为强化学习;后者持续以最优策略生成物替换参考,逐步提高模仿目标,确保指导性而非束缚性。在TOMG-Bench MOLOPT上,匹配三种子评估下,平均SRxSim从GRPO的0.0959、RePO的0.1665提升至0.1773,且在LogP、MR和QED上均获统计显著提升。

原文摘要 · Abstract (English)

Scientific reasoning is an increasingly important capability of large language models, yet improving the robustness and efficiency of training such reasoning remains a key open challenge. We study this problem in instruction-based molecular optimization, where answer-only supervised fine-tuning (SFT) collapses multi-step reasoning and reinforcement learning with verifiable rewards (RLVR) suffers from sparse feedback. Reference-guided Policy Optimization mitigates both by anchoring policy updates to dataset-provided references, but its effectiveness is tightly coupled to reference quality: weak or misaligned references impose a performance ceiling. To overcome this ceiling, we propose active reasoning, a paradigm in which the policy actively decides, on a per-instance basis, when to imitate a reference and when to reinforce its own discoveries, while continuously upgrading what it imitates. We instantiate this paradigm as Active Group Relative Policy Optimization (Active-GRPO), realized through two coupled mechanisms: active imitate-reinforce and active referencing. The former performs imitation learning when the reference still outperforms the policy's own candidates, and shifts to self-improvement via reinforcement learning once the policy has generated molecules that surpass the reference. The latter continuously upgrades the reference itself by replacing it with the best policy-generated candidate discovered so far, progressively raising the imitation target and ensuring that reference guidance remains informative-rather than restrictive-throughout training. Across TOMG-Bench MOLOPT, Active-GRPO improves average SRxSim from 0.0959 for GRPO and 0.1665 for RePO to 0.1773 under matched three-seed evaluation, with statistically significant gains on LogP, MR, and QED.

分子优化自进化推理强化学习主动学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。