构建多语言翻译指令遵循基准,评估模型对格式、术语等多重约束的遵守能力。
IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following

- 设计跨语言多约束测试集,覆盖7种语言与6类约束维度。
- 4506个单约束与2838个多约束样本,采用乘法融合评分避免奖励作弊。
- 揭示大模型在术语与格式约束上表现远低于翻译质量,适合评估专业翻译系统。
现代翻译工作需求已超越语义对等,用户常要求模型保持JSON或HTML结构、遵循术语表、根据上下文消歧,并匹配指定语体,且常需同时满足多项要求。传统指标如BLEU和xCOMET仅衡量语义一致性,难以反映对约束条件的遵守情况;而通用指令遵循基准又忽略了翻译任务的跨语言特性。为此,我们提出IFMTBench,一个涵盖七种语言的多语言翻译指令遵循基准,包含4,506个单约束和2,838个多约束样本,覆盖六类约束维度及五种组合模式,指令以全部七种语言呈现。约束分为可确定性验证的门控子集与基于LLM评分的连续子集,通过乘法规则融合,有效防止奖励黑客。对15个模型的评估发现:指令遵循能力随模型规模增长比翻译质量更显著,术语与结构化格式约束构成主要难度梯度,且通用指令遵循排名与翻译表现相关性较弱。基准代码与数据集已在https://github.com/Tencent-Hunyuan/Hy-MT2/tree/main/IFMTBench发布。
原文摘要 · Abstract (English)
Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated glossaries, disambiguate with provided context, and match prescribed registers, often several at once. Conventional metrics such as BLEU and xCOMET capture semantic fidelity but provide little signal on constraint adherence, while general instruction following benchmarks ignore the cross-lingual nature of translation. We introduce \bench, a benchmark for multilingual translation instruction following covering seven languages, with 4,506 single-constraint and 2,838 multi-constraint items spanning six constraint dimensions and five compositional patterns with instructions issued in all seven languages. Constraints are split into a gating subset verified by deterministic checkers and a continuous subset scored by a rubric-based LLM judge, combined under a multiplicative rule that resists reward hacking. Evaluating 15 models reveals systematic gaps that prior protocols miss: Instruction following scales with size more sharply than translation quality, glossary and structured-format constraints dominate the difficulty gradient, and general instruction following rankings correlate only weakly with translation behavior. Our benchmark are available at https://github.com/Tencent-Hunyuan/Hy-MT2/tree/main/IFMTBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。