分析大模型翻译时如何利用上下文，发现源句比目标句贡献更大。
Analyzing Context Contributions in LLM-based Machine Translation
- 通过对比输入上下文各部分，研究大模型翻译时的注意力分配机制。
- 早期示例贡献更高，且源句比对应目标句影响更大。
- 可检测异常上下文贡献，识别翻译幻觉，适合模型可解释性研究者。
大语言模型在机器翻译中表现优异，能通过少量示例实现上下文学习。然而，其如何利用输入上下文的不同部分仍不清楚。本文全面分析了大模型在机器翻译中对上下文的使用方式，考察了少量示例和源文本等不同部分的作用。研究发现：(1) 少量示例中的源部分贡献大于对应的目标部分，且与翻译方向无关；(2) 使用双语数据微调会改变不同上下文部分的贡献模式；(3) 存在位置偏差，早期示例对生成译文的影响更大。最后，我们证明检查异常的上下文贡献可能揭示病理翻译，如幻觉现象。这些发现揭示了基于大模型的机器翻译内部机制，超越传统编码器-解码器模型的认知。
原文摘要 · Abstract (English)
Large language models (LLMs) have achieved state-of-the-art performance in machine translation (MT) and demonstrated the ability to leverage in-context learning through few-shot examples. However, the mechanisms by which LLMs use different parts of the input context remain largely unexplored. In this work, we provide a comprehensive analysis of context utilization in MT, studying how LLMs use various context parts, such as few-shot examples and the source text, when generating translations. We highlight several key findings: (1) the source part of few-shot examples appears to contribute more than its corresponding targets, irrespective of translation direction; (2) finetuning LLMs with parallel data alters the contribution patterns of different context parts; and (3) there is a positional bias where earlier few-shot examples have higher contributions to the translated sequence. Finally, we demonstrate that inspecting anomalous context contributions can potentially uncover pathological translations, such as hallucinations. Our findings shed light on the internal workings of LLM-based MT which go beyond those known for standard encoder-decoder MT models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。