通过逐标记判断优劣，让模型更可靠地生成新数据。
Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift
- 将生成过程转为逐标记正确性预测，避免直接生成偏差内容。
- 在11个数据集上显著提升分布外泛化能力，优于多种基线方法。
- 学习到的参数可解释，适合作为可控生成的干预工具。
我们提出一种名为逐标记离策略标注（TOPL）的离策略训练范式，将后训练重构为逐标记正确性预测任务。核心思路是：通过训练模型区分回应中的好标记与坏标记，自然引导其生成优质标记，同时规避直接训练模型生成离策略内容所带来的弊端。在文档摘要任务上的实验表明，TOPL在11个数据集上对多种序列级和标记级基线均展现出强大的分布外泛化能力。进一步实验证明，TOPL在机器翻译任务中也具有良好的迁移效果，表明其优势可推广至不同忠实生成任务。消融实验确认，标记级学习信号对性能至关重要，序列级替代方案无法带来类似收益。最后，我们发现TOPL诱导出可解释的模型更新：通过TOPL学习的LoRA适配器可作为线性分类头与方向控制向量使用。
原文摘要 · Abstract (English)
We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。