arXiv:2510.09536cs.CL2025-10ACL被引 8

测试大模型在多语言错字下的表现,发现生成任务最易受影响。

Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors

  • 基于键盘布局模拟真实错字,构建跨语言评测集MulTypo
  • 18个模型在5项任务中均因错字性能下降,推理类任务更脆弱
  • 高资源语言和英译外任务更稳健,适合多语言应用研究者

大型语言模型(LLMs)在多语言真实场景中广泛应用,用户输入常含拼写错误。然而现有评测多假设输入无误,导致模型对跨语言错字的鲁棒性研究不足。为此,我们提出MulTypo,一种基于语言特异性键盘布局和打字行为的多语言错字生成算法。我们在三个模型族、五个下游任务(包括自然语言推理、多选问答、数学推理、机器翻译)上评估了18个开源LLM。结果表明,错字普遍降低性能,尤其在生成任务和需推理的任务中;自然语言推理任务相对更鲁棒。指令微调虽提升干净输入表现,但可能加剧噪声下的脆弱性。观察到语言相关鲁棒性差异:高资源语言普遍比低资源语言更鲁棒,且英译外任务优于英译入任务。研究强调需开展抗噪训练与多语言鲁棒性评估。我们已开源MulTypo Python包及代码,地址为https://github.com/cisnlp/multypo。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly deployed in multilingual, real-world applications with user inputs -- naturally introducing \emph{typographical errors} (typos). Yet most benchmarks assume clean input, leaving the robustness of LLMs to typos across languages largely underexplored. To address this gap, we introduce MulTypo, a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior. We evaluate 18 open-source LLMs across three model families and five downstream tasks spanning language inference, multi-choice question answering, mathematical reasoning, and machine translation tasks. Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring reasoning -- while the natural language inference task is comparatively more robust. Instruction tuning improves clean-input performance but may increase brittleness under noise. We also observe language-dependent robustness: high-resource languages are generally more robust than low-resource ones, and translation from English is more robust than translation into English. Our findings underscore the need for noise-aware training and multilingual robustness evaluation. We release a Python package for MulTypo and make the source code publicly available at https://github.com/cisnlp/multypo.

大模型多语言错字鲁棒性评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。