用重要性加权让微调高效实现多轮交互优化
DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization
- 分离采样与优化：离线生成轨迹，用回报算权重
- 性能媲美强化学习，训练速度如普通微调
- 适合追求高效多轮对话优化的研究者
大型语言模型在多轮交互场景中日益普及，用户或环境可提供轻量反馈。但优化此类行为面临两难：在线强化学习虽能有效处理多轮动态，却因每次更新需生成完整修正轨迹而成本高昂；离线监督微调（SFT）虽高效，却存在分布偏移和行为坍缩问题。为此，我们提出 DRIFT（解耦滚动与重要性加权微调），将理论上的 KL 正则化强化学习目标等价为重要性加权监督学习。DRIFT 通过从固定参考策略离线采样交互轨迹，基于回报计算重要性权重，并在所得数据集上进行加权 SFT 来优化策略。实验表明，DRIFT 在性能上匹配或超越多轮强化学习基线，同时保持标准监督微调的训练效率与简便性。代码已开源。
原文摘要 · Abstract (English)
Large language models are increasingly deployed in multi-turn interactive settings where users or environments can iteratively provide lightweight feedback. Unfortunately, optimizing such behavior presents a sharp dilemma in practice: online reinforcement learning is able to effectively address multi-turn dynamics but is prohibitively expensive due to the cost of generating full correction trajectories at every update, whereas offline supervised fine-tuning (SFT) is efficient but suffers from distribution shift and behavioral collapse. To this end, we novelly propose DRIFT (Decoupled Rollouts and Importance-Weighted Fine-Tuning), a framework that operationalizes the theoretical insight that the KL-regularized RL objective is equivalent to importance-weighted supervised learning. DRIFT decouples rollout from optimization by sampling offline interaction trajectories from a fixed reference policy, deriving return-based importance weights, and optimizing the policy via weighted SFT on the resulting dataset. Empirically, we demonstrate that DRIFT matches or exceeds the performance of multi-turn reinforcement learning baselines while maintaining the training efficiency and simplicity of standard supervised fine-tuning. Code is available at https://github.com/2020-qqtcg/DRIFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。