测试大模型真理解力，而非仅靠长文本记忆
Context Is Not Comprehension
- 将确定性推理任务嵌入叙事伪装中，检验中间步骤理解
- 10,000词后模型准确率从100%暴跌至极低水平
- 适用于各类推理任务，推动下一代模型设计
评估大语言模型的主流方式是考察其从超长输入中回忆显式事实的能力。尽管当前最佳模型在该任务上接近完美，但这掩盖了更难的技能：进行多步推理并追踪未明文出现的中间状态。我们提出冗长列表操作（Verbose ListOps, VLO），将确定性的ListOps计算嵌入叙事伪装中，并支持对每个中间结果进行逐级评估。实验显示，能在原始ListOps中达到约100%准确率的模型，在超过10,000词后便严重崩溃。VLO能定位模型推理链首次偏离的节点，使评估超越单纯上下文长度，转向真正的理解能力。其生成流程与任务无关，可将任意可验证的推理模式——算术、符号、溯因、归纳或可撤销推理——转化为叙事形式，成为未来以推理为中心的模型设计的通用测试平台。
原文摘要 · Abstract (English)
The dominant way of judging Large Language Models (LLMs) has been to ask how well they can recall explicit facts from very long inputs. While today's best models achieve near perfect recall, this masks a harder skill: performing multi-step reasoning and tracking intermediate state that never appears verbatim. We introduce Verbose ListOps (VLO), a benchmark that embeds deterministic ListOps computations inside narrative camouflage and, crucially, allows step-level evaluation of every intermediate result. Experiments show that models which solve raw ListOps with approximately 100% accuracy collapse on VLO after only 10,000 tokens. By exposing where a model's reasoning chain first diverges, VLO moves assessment beyond sheer context length and toward genuine comprehension. VLO's generation pipeline is task-agnostic: it can weave any deterministically verifiable reasoning schema -- arithmetic, symbolic, abductive, inductive or defeasible -- into narrative form. This makes VLO a reusable test-bed for the next wave of reasoning-centric model designs, not merely those with step-explicit scaffolds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。