arXiv:2605.07186cs.CLcs.AI2026-05

发现大模型在文本碎片化时性能先降后升,揭示其认知模式切换的隐藏缺陷。

The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

论文配图:The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
图 1 · 摘自论文原文
  • 通过打乱词边界测试模型信息检索能力
  • 准确率随干扰增加呈倒U型曲线,最低点称'文本诡异谷'
  • 适合关注真实场景下模型鲁棒性的研究者

现有大模型评测多聚焦语法正确的输入,忽视了对不完美文本的评估。本文研究词边界被破坏对大模型信息检测的影响:通过在词中插入空格将词切碎,发现模型检测准确率随插入率上升呈现倒U型曲线,我们称之为‘文本诡异谷’。为解释该现象,提出模式切换假说:模型在近正常文本时使用词级模式,重度碎片化时转为字符级模式,而谷底是两种模式均失效的混乱过渡期。四项实验与一项分析支持该解释:上下文学习无法挽救谷底表现;正则化扰动显著减弱倒U型;数学推理任务中仅Gemini 3.0 Flash出现该现象,表明依赖精确词汇匹配的任务更易受影响;分词熵峰值出现在F1最低点前,符合模式冲突的推断。这些发现揭示了清洁文本评测无法暴露的失败模式,却直接关联于包含噪声或未清洗输入的实际部署场景。

原文摘要 · Abstract (English)

Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect text. In this work, we study how word-boundary corruption affects how LLMs detect targeted information. By inserting whitespace characters within words to break them into fragments, LLMs' detection accuracy follows a U-shaped curve with the increase in insertion rate. We refer to this curve as the Text Uncanny Valley. To explain such observation, we propose a mode transition hypothesis: LLMs operate in a word-level mode for near-normal text and a character-level mode for heavily fragmented text, with the valley marking the disordered transition where neither mode is effective. Four experiments and one analysis are consistent with this account: in-context learning fails to rescue valley-bottom performance; regularizing the perturbation substantially reduces the U-shape; a math reasoning task replicates the U-shape for Gemini 3.0 Flash but not for stronger models, suggesting the effect is attenuated when tasks rely less on exact lexical alignment; and tokenization entropy peaks before the F1 minimum, consistent with a regime-conflict interpretation. These findings reveal a failure mode invisible to clean-text benchmarks yet directly relevant to any deployment scenario involving noisy or uncurated text inputs.

大模型评估文本噪声模式切换

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。