让聊天机器人更懂社交，通过精准优化对话关键段落提升表现
SDPO: Segment-Level Direct Preference Optimization for Social Agents
- 按对话段落动态选重点，比全程优化更精准
- 在SOTOPIA测试中超越GPT-4o等主流模型
- 理论扎实且减少训练噪声，适合社交对话场景
基于大语言模型的社会代理虽能模拟人类社交行为，但在复杂社交对话中表现不足。直接偏好优化（DPO）在对齐人类偏好方面有效，但传统DPO仅关注单轮对话，难以应对多轮互动。现有基于会话级数据的多轮对齐方法虽有潜力，但粒度粗、易引入训练噪声，缺乏理论支撑。为此，我们提出分段级直接偏好优化（SDPO），动态选取交互中的关键段落进行优化，减少训练噪声，并具备严格的理论基础。在SOTOPIA基准上的评估显示，经SDPO调优的代理持续优于现有DPO方法及如GPT-4o等专有模型，彰显其在提升基于LLM的代理社会智能方面的潜力。代码与数据已开源：https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SDPO。
原文摘要 · Abstract (English)
Social agents powered by large language models (LLMs) can simulate human social behaviors but fall short in handling complex social dialogues. Direct Preference Optimization (DPO) has proven effective in aligning LLM behavior with human preferences across various agent tasks. However, standard DPO focuses solely on individual turns, which limits its effectiveness in multi-turn social interactions. Several DPO-based multi-turn alignment methods with session-level data have shown potential in addressing this problem.While these methods consider multiple turns across entire sessions, they are often overly coarse-grained, introducing training noise, and lack robust theoretical support. To resolve these limitations, we propose Segment-Level Direct Preference Optimization (SDPO), which dynamically select key segments within interactions to optimize multi-turn agent behavior. SDPO minimizes training noise and is grounded in a rigorous theoretical framework. Evaluations on the SOTOPIA benchmark demonstrate that SDPO-tuned agents consistently outperform both existing DPO-based methods and proprietary LLMs like GPT-4o, underscoring SDPO's potential to advance the social intelligence of LLM-based agents. We release our code and data at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SDPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。