评测大模型在文化翻译中的表现，发现模型识别与实际运用能力存在明显差距。
Culture-Aware Machine Translation in Large Language Models: Benchmarking and Investigation
- 构建文化敏感的平行语料库CanMT，支持多维度评估。
- 不同翻译策略下模型表现差异显著，文化项类型影响翻译难度。
- 参考译文能大幅提升评测可靠性，对评估文化翻译至关重要。
大型语言模型在通用机器翻译中表现强劲，但在文化敏感场景下的能力仍不明确。为此，我们提出了CanMT——一个面向文化意识翻译的新型驱动平行语料库，并设计了一个理论基础扎实、多维度的评估框架来衡量文化翻译质量。基于CanMT，我们系统评估了多种大模型和翻译系统在不同翻译策略约束下的表现。结果表明，模型间存在显著性能差异，且翻译策略对模型行为有系统性影响。进一步分析显示，不同类型的文化特异性内容翻译难度不同，且模型对文化知识的识别能力与其在译文中正确实现的能力之间仍存在持续差距。此外，引入参考译文可显著提升大模型作为评判者时的评估可靠性，凸显其在文化感知翻译评估中的关键作用。语料库与代码已开源于CanMT。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved strong performance in general machine translation, yet their ability in culture-aware scenarios remains poorly understood. To bridge this gap, we introduce CanMT, a Culture-Aware Novel-Driven Parallel Dataset for Machine Translation, together with a theoretically grounded, multi-dimensional evaluation framework for assessing cultural translation quality. Leveraging CanMT, we systematically evaluate a wide range of LLMs and translation systems under different translation strategy constraints. Our findings reveal substantial performance disparities across models and demonstrate that translation strategies exert a systematic influence on model behavior. Further analysis shows that translation difficulty varies across types of culture-specific items, and that a persistent gap remains between models' recognition of culture-specific knowledge and their ability to correctly operationalize it in translation outputs. In addition, incorporating reference translations is shown to substantially improve evaluation reliability in LLM-as-a-judge, underscoring their essential role in assessing culture-aware translation quality. The corpus and code are available at CanMT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。