测试大模型在无草稿纸情况下连续运算的短期记忆能力
WMF-AM: Probing LLM Working Memory via Depth-Parameterized Cumulative State Tracking
- 通过参数化步骤深度K，精准测量模型累积状态跟踪能力
- 28个模型实测显示，记忆衰减随步骤增加显著恶化
- 适合评估模型在复杂推理中的瓶颈位置，对研究者友好
现有大模型评估使用固定难度基准，无法随模型进步动态调整，且很少分离特定认知过程。本文提出工作记忆保真度-主动操控（WMF-AM），一种对累积状态跟踪能力的探测方法，即在单次查询中无草稿纸地持续维护与更新连续K步操作的中间结果。不同于强调任务编排的多步代理基准，WMF-AM通过参数化深度K，隔离了单轮内的累积负载。核心实验在12个模型家族的28个模型（0.5B至前沿模型）上进行算术累加；匹配的非算术扩展（权限、日程、库存）验证了设计泛化性。三项构造隔离消融实验确认，困难程度主要由累积负载驱动，而非算术能力或实体追踪。我们发布WMF-AM作为轻量级、可重校准的诊断工具，用于刻画模型在累积负载下的性能退化点。代码与数据可在https://github.com/dengzhe-hou/WMF-AM获取。
原文摘要 · Abstract (English)
Existing large language models (LLMs) evaluations use fixed-difficulty benchmarks that cannot adapt as models improve, and rarely isolate specific cognitive processes. We introduce Working Memory Fidelity-Active Manipulation (WMF-AM), a probe of cumulative state tracking, the ability to maintain and update intermediate results across K sequential operations within a single query, without a scratchpad. Unlike multi-step agent benchmarks that stress task orchestration, WMF-AM isolates within-pass cumulative load by parameterizing depth K. The core probe uses arithmetic accumulation on 28 models from 12 families (0.5B to frontier); a matched non-arithmetic extension (permissions, schedules, inventories) confirms the design generalizes beyond arithmetic. Three construct-isolation ablations confirm that cumulative load, not arithmetic skill or entity tracking, drives difficulty. We release WMF-AM as a lightweight, recalibratable diagnostic for characterizing where models degrade under cumulative load. Code and data can be accessed at https://github.com/dengzhe-hou/WMF-AM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。