用在线修正器动态调整约束，让模型在稳定与进步间自动平衡。
Enhancing Reinforcement Learning Fine-Tuning with an Online Refiner
- 引入在线修正器，根据输出质量自动调节约束强度。
- 对话与代码生成任务中，任务奖励显著高于传统方法。
- 适合追求训练稳定性与性能提升的RL微调研究者。
约束对强化学习微调（RFT）的稳定性和防止劣质输出至关重要，但其与优化目标存在内在冲突：更强的约束会限制模型发现更优解的能力。本文提出动态约束机制，基于一个核心洞察：约束仅在出现劣质输出时介入。通过参考模型作为在线修正器，将微调模型的输出转化为最小修正版本，保留正确内容并修复错误；随后使用监督微调损失训练微调模型输出该修正结果。该机制使约束能根据输出质量自动增强或放松。在对话和代码生成任务上的实验表明，动态约束优于KL正则化和无约束基线，在显著提升任务奖励的同时保持训练稳定。
原文摘要 · Abstract (English)
Constraints are essential for stabilizing reinforcement learning fine-tuning (RFT) and preventing degenerate outputs, yet they inherently conflict with the optimization objective because stronger constraints limit the ability of a fine-tuned model to discover better solutions. We propose \textit{dynamic constraints} that resolve this tension by adapting to the evolving capabilities of the fine-tuned model based on the insight that constraints should only intervene when degenerate outputs occur. We implement this by using a reference model as an \textit{online refiner} that takes the response from the fine-tuned model and generates a minimally corrected version which preserves correct content verbatim while fixing errors. A supervised fine-tuning loss then trains the fine-tuned model to produce the refined output. This mechanism yields a constraint that automatically strengthens or relaxes based on output quality. Experiments on dialogue and code generation show that dynamic constraints outperform both KL regularization and unconstrained baselines, achieving substantially higher task rewards while maintaining training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。