arXiv:2510.25187cs.CL2025-10

测试大模型在低资源语言中的理解能力,发现思维链提示效果因模型而异。

Testing Cross-Lingual Text Comprehension In LLMs Using Next Sentence Prediction

  • 用跨语言下一句预测任务评估模型在英语、斯瓦希里语和豪萨语的表现。
  • 模型在豪萨语上准确率下降超40%,LLaMA 3表现最差。
  • 思维链提示对弱模型有帮助,但可能让强模型过度思考导致错误。

尽管大型语言模型在海量数据上训练,但这些数据严重偏向英语。其出色表现是真实理解能力还是数据优势所致?为探究此问题,我们在无法依赖数据丰富的低资源语言环境中进行测试。基于Agarwal等(2025)提出的下一句预测(NSP)测试框架,我们构建了一个大规模基准,涵盖英语(高资源)、斯瓦希里语(中资源)和豪萨语(低资源),每种语言各10,000个问题。测试了GPT-4 Turbo、Gemini 1.5 Flash和LLaMA 3 70B等主流模型。结果表明,所有模型在英语中表现优异,但在斯瓦希里语中准确率下降,在豪萨语中进一步骤降,其中LLaMA 3表现最差。引入思维链(CoT)提示后,对表现较弱的LLaMA 3有显著提升作用;然而,对于更强大的GPT-4和Gemini,同一方法常引发“过度思考”,反而降低跨语言场景下的准确率。这说明思维链并非通用解法,其有效性高度依赖模型基线能力与任务上下文。本研究揭示了大模型在跨语言理解中的薄弱环节,明确了思维链在不同情境下的利弊,为评估与优化跨语言性能提供了关键框架。

原文摘要 · Abstract (English)

While large language models are trained on massive datasets, this data is heavily skewed towards English. Does their impressive performance reflect genuine ability or just this data advantage? To find out, we tested them in a setting where they could not rely on data abundance: low-resource languages. Building on prior work Agarwal et al. (2025) that used Next Sentence Prediction (NSP) as a test, we created a large-scale benchmark with 10,000 questions each for English (a high-resource language), Swahili (medium-resource), and Hausa (low-resource). We then tested several top models, including GPT-4 Turbo, Gemini 1.5 Flash, and LLaMA 3 70B, to see how their performance holds up. The results painted a clear picture of how levels of language resources impact outcomes. While all models excelled in English, their accuracy dropped in Swahili and fell sharply in Hausa, with LLaMA 3 struggling the most. The story became even more interesting when we introduced Chain-of-Thought (CoT) prompting. For the struggling LLaMA 3, CoT acted as a helpful guide, significantly boosting its accuracy. However, for the more capable GPT-4 and Gemini, the same technique often backfired, leading to a kind of "overthinking" that hurt their results in the cross-lingual context. This reveals that Chain-of-Thought is not a universal solution; its effectiveness depends heavily on the model's baseline capability and the specific context of the task. Our framework pinpoints LLM weaknesses, highlights when CoT helps or hinders cross-lingual NSP performance, and factors influencing their decisions.

大模型跨语言思维链评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。