测试大模型对混语文本的理解能力，发现嵌入外语反而提升效果。
Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text
- 构建混语版本的评测基准，系统评估大模型理解力。
- 外语词汇干扰英语时模型性能下降，但嵌入外语可提升表现。
- 微调比提示工程更有效，适合需要稳定处理混语的应用场景。
代码切换（CSW）指在单次话语中交替使用两种或更多语言，广泛存在于多语言社群及在线内容中。随着大型语言模型（LLMs）在内容处理与生成中的核心地位日益凸显，其常需面对混语输入。本文通过生成现有推理与理解基准的混语变体，系统评估了LLMs在代码切换情境下的理解能力。结果表明，当外语词汇打断英语文本时，模型性能显著下降——即便存在语言约束亦然；而将英语嵌入其他语言却常能提升理解效果。尽管提示工程效果不一，但微调展现出更稳定的性能改善潜力，是缓解理解退化的可靠路径。
原文摘要 · Abstract (English)
Code-switching (CSW) is the act of alternating between two or more languages within a single discourse. This phenomenon is widespread in multilingual communities, and increasingly prevalent in online content, where users naturally mix languages in everyday communication. As a result, Large Language Models (LLMs), now central to content processing and generation, are frequently exposed to code-switched inputs. Given their widespread use, it is crucial to understand how LLMs process and reason about such mixed-language text. This paper presents a systematic evaluation of LLM comprehension under code-switching by generating CSW variants of established reasoning and comprehension benchmarks. While degradation is evident when foreign tokens disrupt English text$\unicode{x2013}$even under linguistic constraints$\unicode{x2013}$embedding English into other languages often improves comprehension. Though prompting yields mixed results, fine-tuning offers a more stable path to degradation mitigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。