arXiv:2604.08571cs.LGcs.AI2026-04被引 2

测试大模型在文字变形下的推理鲁棒性,发现多数模型易受干扰。

Robust Reasoning Benchmark

  • 对数学题文本施加13种变形,评估模型抗干扰能力
  • 开源模型在扰动下准确率最高下降54%,部分任务全崩
  • 发现推理过程会污染注意力机制,需内置上下文重置

尽管大型语言模型在标准数学基准上表现优异,但其解题能力受上下文和文本格式影响。我们提出鲁棒推理基准(RRB),对AIME 2024和AIME 2025的题目应用13种确定性文本扰动。评估8个前沿模型后发现,主流模型总体稳健,唯独Claude对多数变形提示完全拒绝响应。开源权重推理模型在结构噪声下呈现多种失效模式(认知震荡、分词崩溃、推理坍缩),平均准确率下降达54%,个别任务下降100%。进一步研究发现:在单个上下文窗口内连续求解多个独立数学问题时,模型自身思维链引发注意力稀释(Intra-Query Attention Dilution)。7B至120B参数的开源模型均出现后续问题准确率下降,表明中间推理步骤逐步污染密集注意力机制。我们认为,要实现可靠推理,未来架构需在模型自有的思维链中集成显式上下文重置,由此引出关于推理任务最优粒度的开放研究问题。

原文摘要 · Abstract (English)

While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their problem-solving abilities depend on the context and textual formatting. We introduce the Robust Reasoning Benchmark (RRB), a pipeline of 13 deterministic textual perturbations applied to AIME 2024 and AIME 2025. Evaluating 8 state-of-the-art models, we find that frontier models are largely resilient, with the notable exception of Claude, which categorically refuses many transformed prompts. Open-weights reasoning models exhibit a range of failure modes under structural noise (cognitive thrashing, tokenization breakdown, and reasoning collapse), with up to 54% average accuracy drops across perturbations and up to 100% on some. We further study one of these failure modes in isolation: attention dilution caused by the model's own chain-of-thought. By tasking models with solving multiple independent mathematical problems sequentially within a single context window, we identify Intra-Query Attention Dilution. Open-weights models ranging from 7B to 120B parameters exhibit accuracy decay on subsequent problems, suggesting that intermediate reasoning steps progressively pollute standard dense attention mechanisms. We argue that in order to achieve reliable reasoning, future architectures need to integrate explicit contextual resets within models' own chain-of-thought, leading to open research questions regarding the optimal granularity of reasoning tasks.

推理鲁棒性注意力机制数学推理模型评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。