用奖励加权微调优化对话策略，提升问答质量与奖励得分。
Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization
- 将离线强化学习转为奖励加权微调，沿用监督微调技术。
- 在短时问答任务中，奖励与语言质量均显著优于现有方法。
- 适合需要高效优化对话策略的研究者和工程师。
离线强化学习（Offline RL）指从预先收集的轨迹和奖励数据中学习策略。本文提出一种适用于大语言模型（LLMs）的实用离线强化学习方法，将问题重构为奖励加权微调，可采用与监督微调（SFT）相似的技术求解。为验证该方法价值，我们将其应用于固定长度的短时问答策略学习，使智能体能够推理潜在答案或提出澄清问题。相比当前主流方法（基于SFT与直接偏好优化），本方法无需额外超参数，且直接优化奖励。实证对比显示，在优化奖励和语言质量上均有显著提升。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) is a variant of RL where the policy is learned from a previously collected dataset of trajectories and rewards. In our work, we propose a practical approach to offline RL with large language models (LLMs). We recast the problem as reward-weighted fine-tuning, which can be solved using similar techniques to supervised fine-tuning (SFT). To showcase the value of our approach, we apply it to learning short-horizon question-answering policies of a fixed length, where the agent reasons about potential answers or asks clarifying questions. Our work stands in a stark contrast to state-of-the-art methods in this domain, based on SFT and direct preference optimization, which have additional hyper-parameters and do not directly optimize for rewards. We compare to them empirically, and report major gains in both optimized rewards and language quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。