用可控干扰项测试大模型推理鲁棒性，发现干扰严重影响判断。
How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark
- 构建带精准干扰项的符号推理数据集，实现可复现评估
- 模型在含干扰时推理路径与计算准确率显著下降
- 通过分步树搜索优化推理，提升复杂场景下的稳定性
我们提出GSM-DC——一个用于评估大语言模型（LLMs）在系统性无关上下文（IC）干扰下推理鲁棒性的合成基准。该数据集通过精确注入干扰项构建符号推理图，支持严谨、可复现的评估。实验表明，LLMs对无关上下文高度敏感，影响推理路径选择和算术准确性。此外，使用强干扰项训练模型，能提升其在分布内与分布外场景的表现。我们还提出一种基于过程奖励模型引导的分步树搜索方法，在分布外条件下显著增强模型鲁棒性。
原文摘要 · Abstract (English)
We introduce Grade School Math with Distracting Context (GSM-DC), a synthetic benchmark to evaluate Large Language Models' (LLMs) reasoning robustness against systematically controlled irrelevant context (IC). GSM-DC constructs symbolic reasoning graphs with precise distractor injections, enabling rigorous, reproducible evaluation. Our experiments demonstrate that LLMs are significantly sensitive to IC, affecting both reasoning path selection and arithmetic accuracy. Additionally, training models with strong distractors improves performance in both in-distribution and out-of-distribution scenarios. We further propose a stepwise tree search guided by a process reward model, which notably enhances robustness in out-of-distribution conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。