让智能体通过多步交互理解用户偏好，自动优化个性化服务。
ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning
- 按整个交互过程统一优化，而非逐步独立决策。
- 在长任务中显著提升个性化推理准确率，超越通用模型。
- 适合需要持续理解用户意图的复杂交互场景。
智能体在真实环境中的能力迅速提升，能与外部工具互动并为用户提供服务。然而，相较于明确指令的任务，以用户为中心的场景往往存在模糊请求，导致解空间极大。因此，理解用户的个性化偏好对缩小候选解空间至关重要。这带来了个性化智能体推理的新挑战：智能体需同时与用户和环境交互以提供定制化服务。本文提出ODYSSE，一种用于个性化智能体推理的强化微调框架。核心是提出分幕式组相对策略优化（ESPO），作为组相对策略优化（GRPO）的新扩展，旨在应对个性化智能体推理中长动作序列与强跨步依赖的问题。与独立优化每一步不同，ESPO引入基于整幕的奖励机制与幕内优势估计，使上游信息有效引导下游个性化决策，支持智能体在多步交互中逐步解析模糊用户请求。我们还设计了幕内批量采样器，将同一幕中的动作合并成统一训练批次，促进在ESPO下的连贯优化。在真实的长时序个性化图形界面推理任务上评估显示，ODYSSE始终优于专业与通用视觉语言模型，验证了其在个性化智能体推理中的有效性。
原文摘要 · Abstract (English)
Agentic systems have rapidly advanced in their ability to interact with real-world environments, leverage external tools, and provide services for users. However, unlike natural-world tasks that assume well-defined instructions, human-centered scenarios are characterized by ambiguous requests that lead to large, open-ended solution spaces. Decoding users' personalized preferences is therefore essential for narrowing the candidate solution space. This introduces a new challenge, personalized agentic reasoning, which requires agents to jointly interact with both users and environments to deliver personalized services. In this paper, we present ODYSSE, a Reinforced Fine-Tuning (RFT) framework for personalized agentic reasoning. At its core, ODYSSE proposes Episode-wise GRPO (ESPO), a novel extension of Group Relative Policy Optimization (GRPO) designed to address long action horizons and strong cross-step dependencies in personalized agentic reasoning. Rather than optimizing individual steps independently, ESPO introduces an episode-level reward mechanism together with episodic advantage estimation, enabling upstream evidence to effectively guide downstream personalized decisions and allowing agents to progressively resolve ambiguous user requests across multiple interaction steps. We further propose an episodic batch sampler that groups actions from the same episode into unified training batches, facilitating coherent optimization under ESPO. We evaluate ODYSSE on realistic long-horizon personalized GUI reasoning tasks. Experimental results demonstrate that ODYSSE consistently outperforms both specialist and general-purpose LVLMs, highlighting its effectiveness for personalized agentic reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。