让智能体学会根据用户偏好个性化决策,提升真实场景下的适应能力。
From Correctness to Preference: A Framework for Personalized Agentic Reinforcement Learning

- 分离通用任务奖励与用户偏好奖励,用用户锚点稳定学习过程。
- 在多个基准上超越现有方法,尤其在复杂任务中表现更优。
- 适合需要个性化交互的智能体应用,如客服、助手等场景。
代理强化学习在具有明确成功信号的任务中已取得显著进展。然而,许多真实世界的应用需要用户定制行为:相同查询在不同用户间可能需要不同的规划策略和工具使用决策。这一设定带来关键挑战:通用奖励无法捕捉异质用户偏好,观察到的行为被从众效应纠缠,且扁平记忆难以支持个性化技能检索。为此,我们提出一个统一的个性化代理强化学习框架,将个性化融入训练阶段优化。核心是个性化锚点奖励解耦策略优化(PARPO),该方法将通用任务质量奖励与个性化偏好奖励解耦,并利用用户特定锚点在异构奖励尺度下稳定学习。我们进一步引入两阶段偏好解耦奖励模型,以及偏好对齐技能演化图记忆(PSGM),实现个性化监督与偏好对齐的技能检索。三者共同构成偏好识别、策略优化与结构化技能积累的闭环。在ETAPP、ETAPP-Hard和SJAgent上的实验表明,本框架持续优于强基线的记忆与强化学习方法。代码与数据包含在补充材料中。
原文摘要 · Abstract (English)
Agentic reinforcement learning (Agentic RL) has achieved strong progress in tasks with clear success signals. However, many real-world agent applications require user-conditioned behavior: the same query may call for different planning strategies and tool-use decisions across users. This setting raises key challenges: generic rewards cannot capture heterogeneous user preferences, observed behaviors are entangled with conformity effects, and flat memories cannot support personalized skill retrieval. To this end, we propose a unified personalized Agentic RL framework that embeds personalization into training-time optimization. At its core is \emph{Personalized Anchor Reward-Decoupled Policy Optimization} (\textbf{PARPO}), which decouples generic task-quality rewards from personalized preference rewards and uses user-specific anchors to stabilize learning under heterogeneous reward scales. We further introduce a two-stage preference-disentangled reward model and \emph{Preference-Aligned Skill Evolution Graph Memory} (\textbf{PSGM}) for personalized supervision and preference-aligned skill retrieval. Together, they form a closed loop of preference identification, policy optimization, and structured skill accumulation. Experiments on ETAPP, ETAPP-Hard, and SJAgent show that our framework consistently outperforms strong memory and RL baselines. Code and data are included in the supplementary materials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。