用教师模型动态生成反馈，让机器翻译逐步优化。
RL from Teacher-Model Refinement: Gradual Imitation Learning for Machine Translation
- 教师模型对候选译文做微调，生成可学习的强化信号
- 在FLORES-200上超越SFT、DPO等基线，提升语义与实体保留
- 无需人工偏好数据，适合大规模翻译优化场景
针对机器翻译中的偏好学习方法（如DPO）依赖大量人工标注偏好三元组且泛化能力差的问题，本文提出强化学习从教师模型精炼（RLfR）。该方法将静态三元组替换为由冻结教师模型生成的在线策略、以行为条件的局部微调。每步中，智能体采样候选译文，教师进行最小化局部编辑，智能体通过复合奖励被强化：该奖励结合缩放后的负编辑距离（用于词汇与结构准确率）和COMET（用于语义充分性）。此机制在不依赖显式偏好数据的前提下，提供稳定且模型感知的学习信号。在FLORES-200（英→德、西、中、韩、日）上的实验表明，RLfR持续优于强基线（MT-SFT、DPO、固定参考强化学习），显著提升语义质量与实体保留，并在大模型判别评估中表现更优。
原文摘要 · Abstract (English)
Preference-learning methods for machine translation (MT), such as Direct Preference Optimization (DPO), have shown strong gains but typically rely on large, carefully curated preference triplets and often struggle to generalize beyond their tuning domains. We propose Reinforcement Learning from Teacher-Model Refinement (RLfR), which replaces static triplets with on-policy, actor-conditioned refinements produced by a frozen teacher. At each step, the actor samples candidate translations, the teacher performs a minimal local edit of each draft, and the actor is reinforced to close the gap using a composite reward that combines scaled negative edit distance for lexical and structural fidelity with COMET for semantic adequacy. This formulation yields a stable, model-aware learning signal without requiring explicit preference datasets. Experiments on FLORES-200 (English to German, Spanish, Chinese, Korean, and Japanese) show that RLfR consistently outperforms strong MT-SFT, DPO, and fixed-reference RL baselines, improving semantic quality and entity preservation, and also achieves superior performance under LLM-based judge evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。