针对迭代优化任务，提出细粒度的回合级强化学习方法。
TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimization
- 基于回合级分组采样，实现对每轮交互的精细优化。
- 在电路尺寸设计任务中，超越标准GRPO与贝叶斯优化方法。
- 适用于需多轮迭代且依赖领域知识的科学优化场景。
大语言模型通过工具集成在复杂任务中展现出强大的推理能力，通常被建模为马尔可夫决策过程，并使用轨迹级强化学习算法（如GRPO）进行优化。然而，一类典型的推理任务——迭代优化，面临独特挑战：代理在多轮交互中面对相同的环境状态，且轨迹价值由最佳回合奖励决定，而非累计回报。现有基于GRPO的方法无法在该设定下实现细粒度的回合级优化，而黑箱优化方法则忽略了先验知识与推理能力。为此，我们提出回合级GRPO（TL-GRPO），一种轻量级强化学习算法，通过回合级分组采样实现细粒度优化。我们在模拟电路尺寸设计（ACS）这一需要多次仿真与领域专长的复杂科学优化任务上评估了TL-GRPO。结果表明，TL-GRPO在多种规格下均优于标准GRPO和贝叶斯优化方法。此外，使用TL-GRPO训练的30B模型在相同仿真预算下达到了当前最优性能，展现了出色的泛化能力与实际应用价值。
原文摘要 · Abstract (English)
Large language models have demonstrated strong reasoning capabilities in complex tasks through tool integration, which is typically framed as a Markov Decision Process and optimized with trajectory-level RL algorithms such as GRPO. However, a common class of reasoning tasks, iterative optimization, presents distinct challenges: the agent interacts with the same underlying environment state across turns, and the value of a trajectory is determined by the best turn-level reward rather than cumulative returns. Existing GRPO-based methods cannot perform fine-grained, turn-level optimization in such settings, while black-box optimization methods discard prior knowledge and reasoning capabilities. To address this gap, we propose Turn-Level GRPO (TL-GRPO), a lightweight RL algorithm that performs turn-level group sampling for fine-grained optimization. We evaluate TL-GRPO on analog circuit sizing (ACS), a challenging scientific optimization task requiring multiple simulations and domain expertise. Results show that TL-GRPO outperforms standard GRPO and Bayesian optimization methods across various specifications. Furthermore, our 30B model trained with TL-GRPO achieves state-of-the-art performance on ACS tasks under same simulation budget, demonstrating both strong generalization and practical utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。