arXiv:2601.22491cs.CL2026-01

用甜点区思想让智能体更高效找到最优解

SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization

  • 按层级逐步增强奖励,引导智能体聚焦最优解区域
  • 在12个任务上提升2.5倍采样效率,跨任务迁移能力强
  • 适合需要精细优化路径的复杂决策场景

基于可验证奖励的强化学习已成为训练智能体的强大范式。然而,现有方法多采用二值奖励,无法区分达成相同结果的轨迹间质量差异,从而忽略解空间内的潜在多样性。受网球击球甜点区概念启发,我们提出甜点区学习(SSL)框架,为智能体优化提供差异化引导。SSL遵循简单有效的原则:通过逐级放大的分层奖励,引导策略向解空间的甜点区靠近。该原则可自然适配多种任务:视觉感知任务利用距离分层建模奖励接近度,复杂推理任务则奖励向有前景解的渐进进展。理论证明SSL保持最优解排序并提升梯度信噪比,促进更定向的优化。在GUI感知、短/长时规划及复杂推理任务上的广泛实验表明,相比强基线在12个基准上持续提升,采样效率最高达2.5倍增长,且具备有效跨任务迁移能力。本工作确立了SSL作为训练高效稳健智能体的通用原则。

原文摘要 · Abstract (English)

Reinforcement learning with verifiable rewards has emerged as a powerful paradigm for training intelligent agents. However, existing methods typically employ binary rewards that fail to capture quality differences among trajectories achieving identical outcomes, thereby overlooking potential diversity within the solution space. Inspired by the ``sweet spot'' concept in tennis-the racket's core region that produces optimal hitting effects, we introduce \textbf{S}weet \textbf{S}pot \textbf{L}earning (\textbf{SSL}), a novel framework that provides differentiated guidance for agent optimization. SSL follows a simple yet effective principle: progressively amplified, tiered rewards guide policies toward the sweet-spot region of the solution space. This principle naturally adapts across diverse tasks: visual perception tasks leverage distance-tiered modeling to reward proximity, while complex reasoning tasks reward incremental progress toward promising solutions. We theoretically demonstrate that SSL preserves optimal solution ordering and enhances the gradient signal-to-noise ratio, thereby fostering more directed optimization. Extensive experiments across GUI perception, short/long-term planning, and complex reasoning tasks show consistent improvements over strong baselines on 12 benchmarks, achieving up to 2.5X sample efficiency gains and effective cross-task transferability. Our work establishes SSL as a general principle for training capable and robust agents.

强化学习智能体优化奖励设计采样效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。