arXiv:2607.26391cs.LGq-bio.BM2026-07

用动作价值引导分子生成,提升优化效果。

Q-Steer: Action-Value Guidance for Molecular Policy Optimization

论文配图:Q-Steer: Action-Value Guidance for Molecular Policy Optimization
图 1 · 摘自论文原文
  • 在生成过程中实时使用预训练价值评分器指导下一步选择
  • 固定10000次调用预算下,8个组合中均提升有效唯一分子得分
  • 适合各类分子生成模型和优化器,无需改变原有流程

基于目标函数的分子优化仅在完整分子生成后提供奖励,而每次采样需做出多次局部的下一步选择。这种延迟反馈使优化过程变得短视:优化器知道分子好,却不知哪些中间操作促成了这一结果。我们提出Q-Steer,一种面向分子语言模型的滚动期动作价值引导机制。Q-Steer利用离线训练且冻结的前缀-动作价值评分器PAVS-Q,估算在给定部分SMILES前缀下采用候选下一个符号的下游奖励,并将归一化后的价值奖励加入采样概率。优化器更新规则与在线调用预算保持不变;主张是在固定在线调用预算下的性能提升,而非总计算量相等。在PMO23数据集上,固定10,000次在线调用预算下,对两种分子语言模型主干与四种优化器进行完整因子实验,结果显示在全部八个主干-优化器组合中,Q-Steer均提升平均有效唯一分数,宏观平均得分提升在+0.033至+0.049之间,每组有18-20项任务表现更优。机制控制实验表明动作身份至关重要:广播前缀值接近中性,而打乱动作值会损害性能。这些结果支持Q-Steer作为可复用的滚动期动作价值封装器,能在不改变在线调用预算的前提下,提升各类优化器家族与策略主干的平均分子优化奖励。

原文摘要 · Abstract (English)

Oracle-limited molecular optimization gives reward only after a complete molecule is generated, while each rollout requires many local next-token decisions. This delayed-feedback interface makes molecular policy optimization myopic: an optimizer can learn that a molecule was good without knowing which intermediate actions made it good. We introduce Q-Steer, a rollout-time action-value steering primitive for molecular language models. Q-Steer uses an offline-trained and frozen prefix-action value scorer, PAVS-Q, that estimates the downstream reward of taking a candidate next token under a partial SMILES prefix, then adds a normalized value bonus to sampling logits. The optimizer update rule and online oracle budget are unchanged; the claim is fixed-online-oracle performance, not equal total compute. On PMO23 with a fixed 10,000-call online budget, complete factorial studies across two molecular language-model backbones and four optimizers show that Q-Steer improves mean valid-unique score in all eight backbone-optimizer cells, with positive macro mean-score gains between +0.033 and +0.049 and 18-20 task wins per cell. Mechanism controls show that action identity matters: prefix-broadcast values are nearly neutral, while shuffled action values harm performance. These results support Q-Steer as a reusable rollout-time action-value wrapper that improves average molecular optimization reward across optimizer families and policy backbones without changing the online oracle budget.

分子生成强化学习动作价值SMILES

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。