用大模型直接翻译整篇文档，效果更好但不能靠BLEU判断。
Fine-Grained and Multi-Dimensional Metrics for Document-Level Machine Translation
- 直接让指令调优大模型一次性翻译全文，无需额外训练。
- 相比逐句翻译，整体流畅性和连贯性显著提升。
- 建议用GPT-4评估文档一致性，避免依赖误导性的BLEU分数。
大型语言模型（LLMs）在自然语言处理任务中表现卓越，包括机器翻译（MT），但多数研究聚焦于句子级翻译。本文探究指令调优的LLM在文档级翻译（docMT）中的内在能力。不同于以往需要专门技术的方法，我们通过直接提示模型一次性翻译整篇文档进行评估。结果表明，即使未进行文档级微调，该方法也能提升翻译质量，优于逐句翻译。然而，这一优势在BLEU得分上并未体现，因BLEU常偏好句子级翻译。为此，我们提出使用“大模型作为裁判”（LLM-as-a-judge）的评估范式，利用GPT-4更细致地评判文档的连贯性、准确性和流畅性。研究表明，指令调优的LLM能有效利用文档上下文进行翻译。但我们警告，不宜使用BLEU评估文档级翻译，因其常产生误导性结果。代码与GPT-4评估输出已开源于https://github.com/EIT-NLP/BLEUless_DocMT。
原文摘要 · Abstract (English)
Large language models (LLMs) have excelled in various NLP tasks, including machine translation (MT), yet most studies focus on sentence-level translation. This work investigates the inherent capability of instruction-tuned LLMs for document-level translation (docMT). Unlike prior approaches that require specialized techniques, we evaluate LLMs by directly prompting them to translate entire documents in a single pass. Our results show that this method improves translation quality compared to translating sentences separately, even without document-level fine-tuning. However, this advantage is not reflected in BLEU scores, which often favor sentence-based translations. We propose using the LLM-as-a-judge paradigm for evaluation, where GPT-4 is used to assess document coherence, accuracy, and fluency in a more nuanced way than n-gram-based metrics. Overall, our work demonstrates that instruction-tuned LLMs can effectively leverage document context for translation. However, we caution against using BLEU scores for evaluating docMT, as they often provide misleading outcomes, failing to capture the quality of document-level translation. Code and the outputs from GPT4-as-a-judge are available at https://github.com/EIT-NLP/BLEUless_DocMT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。