揭露检索增强翻译在噪声下的脆弱性,发现低资源语言更易出错。
Exposing the Cracks: Vulnerabilities of Retrieval-Augmented LLM-based Machine Translation
- 构建噪声合成框架与评估指标,系统测试检索增强翻译鲁棒性。
- 低资源语言对噪声敏感,翻译常失真;大模型反而更易被错误信息误导。
- 注意力偏移至噪声内容却仍自信,提示需自验证机制设计。
检索增强型大语言模型机器翻译(REAL-MT)在需要知识的任务如习语翻译中表现良好,但在实际部署中检索结果常含噪声,其可靠性尚不明确。为此,本文提出一种噪声合成框架与新评估指标,系统评估REAL-MT的鲁棒性。采用Qwen系列模型(包括标准LLM与增强推理的大模型LRM),在高、中、低资源语言对上测试习语翻译性能。结果表明,低资源语言对更依赖检索内容,在噪声下性能下降更严重,常产生无意义翻译;尽管LRM具备更强推理能力,却无法纠正错误,甚至更易被噪声误导,因注意力转移至噪声内容而自信度上升,显示严重校准偏差。进一步探索训练无关与微调策略,虽可提升抗噪性但牺牲干净环境下的性能,揭示根本性权衡。研究凸显当前方法局限,强调需引入自验证集成机制。
原文摘要 · Abstract (English)
\textbf{RE}trieval-\textbf{A}ugmented \textbf{L}LM-based \textbf{M}achine \textbf{T}ranslation (REAL-MT) shows promise for knowledge-intensive tasks like idiomatic translation, but its reliability under noisy retrieval contexts remains poorly understood despite this being a common challenge in real-world deployment. To address this gap, we propose a noise synthesis framework and new metrics to evaluate the robustness of REAL-MT systematically. Using this framework, we instantiate REAL-MT with Qwen-series models, including standard LLMs and large reasoning models (LRMs) with enhanced reasoning, and evaluate their performance on idiomatic translation across high-, medium-, and low-resource language pairs under synthesized noise. Our results show that low-resource language pairs, which rely more heavily on retrieved context, degrade more severely under noise than high-resource ones and often produce nonsensical translations. Although LRMs possess enhanced reasoning capabilities, they show no improvement in error correction and are even more susceptible to noise, tending to rationalize incorrect contexts. We find that this stems from an attention shift away from the source idiom to noisy content, while confidence increases despite declining accuracy, indicating poor calibration. To mitigate these issues, we investigate training-free and fine-tuning strategies, which improve robustness at the cost of performance in clean contexts, revealing a fundamental trade-off. Our findings highlight the limitations of current approaches, underscoring the need for self-verifying integration mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。