通过逐词优化，解决大模型多语言输出混乱问题
TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
- 在生成过程中定位易错位置，逐词调整输出策略
- 在多种语言模型上显著提升语言一致性，不降低任务准确率
- 适合需要稳定多语言输出的场景，如跨语言客服、翻译
大语言模型虽具备强大多语言能力，但常无法稳定输出目标语言，出现语言混淆现象。以往基于序列级微调的方法（如DPO、ORPO、GRPO）在整条回复层面操作，可能导致模型通用能力下降。为此，我们提出逐词策略优化（TLPO），通过识别易错位置，探索候选词并采用定制目标函数，在细粒度层面抑制错误输出，实现针对性干预。实验表明，TLPO在多个多语言LLM和多种语言上显著优于基线方法，在提升语言一致性的同时保持下游任务准确性。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate strong multilingual capabilities, yet often fail to consistently generate responses in the intended language, exhibiting a phenomenon known as language confusion. Prior mitigation approaches based on sequence-level fine-tuning, such as DPO, ORPO, and GRPO, operate at the level of entire responses and can lead to unintended degradation of general model capabilities, motivating the need for more fine-grained alternatives. To address this, we introduce Token-Level Policy Optimization (TLPO), a fine-tuning framework designed to mitigate language confusion through localized, token-level updates. TLPO identifies error-prone positions, explores alternative candidate tokens, and updates the policy using a tailored objective to suppress error-inducing outputs at a granular level. This selective intervention enables effective mitigation of language confusion without compromising the model's general abilities. Experiments on multiple multilingual LLMs across diverse languages demonstrate that TLPO significantly outperforms baselines in improving language consistency while preserving downstream task accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。