用大模型从一段对话反馈中自动拆解出每轮奖励，提升对话质量。
Aligning Dialogue Agents with Global Feedback via Large Language Model Multimodal Reward Decomposition
- 用预训练大模型分析对话整体反馈，推断每轮对话的隐含奖励。
- 结合语音语调、眼神、表情等行为信息，提升奖励分解精度。
- 无需人工标注奖励，适合需要快速优化对话系统的场景。
我们提出一种基于大语言模型的奖励分解框架，仅需单次会话级反馈即可对齐对话智能体。利用冻结的预训练大语言模型（LLM）的推理能力，将全局会话反馈分解为细粒度的局部隐式奖励。第一种文本仅变体通过对话文本提示LLM进行奖励分解；第二种多模态变体额外引入语音语调、视线方向和面部情绪等行为线索，并以自然语言描述形式融入。这些推断出的回合级奖励被提炼为轻量级奖励模型，用于强化学习微调对话生成。在与现有最优奖励分解方法对比中，两种变体均在人类评估中显著提升对话质量，表明大语言模型是强大的奖励分解器，可替代繁琐的人工奖励设计与细粒度人工反馈。
原文摘要 · Abstract (English)
We propose a large language model based reward decomposition framework for aligning dialogue agents using only a single session-level feedback signal. We leverage the reasoning capabilities of a frozen, pretrained large language model (LLM) to infer fine-grained local implicit rewards by decomposing global, session-level feedback. Our first \emph{text-only} variant prompts the LLM to perform reward decomposition using only the dialogue transcript. The second \emph{multimodal} variant incorporates additional behavioral cues, such as pitch, gaze, and facial affect, expressed as natural language descriptions. These inferred turn-level rewards are distilled into a lightweight reward model, which we utilize for RL-based fine-tuning for dialogue generation. We evaluate both text-only and multimodal variants against state-of-the-art reward decomposition methods and demonstrate notable improvements in human evaluations of conversation quality, suggesting that LLMs are strong reward decomposers that obviate the need for manual reward shaping and granular human feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。