用偏好对齐提升翻译质量？实验证明它不总是靠谱。
Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
- 用对比偏好优化（CPO）直接调整模型权重以匹配评分器偏好。
- CPO在高质量数据上优于监督微调，但下游指标波动大。
- 仅用基础模型生成译文，效果接近多系统融合且更稳定。
神经评分器因其与人类判断高度相关，正逐步取代传统词汇类指标成为机器翻译评估主流。研究者因此采用基于质量的解码策略，在性能上超越了基于似然的方法。随着大语言模型（LLM）兴起，偏好对齐技术受到关注，其通过直接优化模型权重来匹配由质量评估器诱导的偏好，有望提升翻译质量。本文聚焦对比偏好优化（CPO），开展大量实验评估其对翻译质量的影响。结果表明：尽管在高质量数据上CPO始终优于监督微调（SFT）的对齐指标表现，但其在下游评估指标间存在不稳定性，尤其在神经评分器与词汇类指标之间差异显著；此外，我们证明仅依赖基座模型生成候选译文，即可达到与使用多个外部系统相当的性能，同时保持更强的跨下游指标一致性。
原文摘要 · Abstract (English)
Neural metrics for machine translation (MT) evaluation have become increasingly prominent due to their superior correlation with human judgments compared to traditional lexical metrics. Researchers have therefore utilized neural metrics through quality-informed decoding strategies, achieving better results than likelihood-based methods. With the rise of Large Language Models (LLMs), preference-based alignment techniques have gained attention for their potential to enhance translation quality by optimizing model weights directly on preferences induced by quality estimators. This study focuses on Contrastive Preference Optimization (CPO) and conducts extensive experiments to evaluate the impact of preference-based alignment on translation quality. Our findings indicate that while CPO consistently outperforms Supervised Fine-Tuning (SFT) on high-quality data with regard to the alignment metric, it may lead to instability across downstream evaluation metrics, particularly between neural and lexical ones. Additionally, we demonstrate that relying solely on the base model for generating candidate translations achieves performance comparable to using multiple external systems, while ensuring better consistency across downstream metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。