提出黑盒审计方法,低成本检测大模型对时间相关决策的隐性记忆。
HindsightBench: A Black-Box Behavioral Audit Protocol for Parametric Hindsight in Time-Indexed LLM Decision Tasks

- 通过日期操控矩阵与双探针设计,实现无回测、无日志的黑盒审计。
- 发现2026年生成模型普遍具备时间触发反射,而2024年模型多缺失此特性。
- 量化方案与推理模式会影响审计结果,需纳入协议约束条件。
大型语言模型会将历史日期之后发生的事件信息以参数形式嵌入到基于该日期的决策任务中——并非直接查询实际结果,而是对时期背景的知识性记忆。尽管这种现象已确认存在,但用户缺乏低成本的审计手段。本文提出 HindsightBench,一种黑盒审计协议,可在探针级别成本下(无需回测、无需对数概率、无需语料库访问)评估任意时间索引的LLM决策任务中的参数化事后认知。该协议结合四维日期操控矩阵(揭示/仅日期/遮蔽/移植)、双记忆探针(日期恢复;结果回忆)及六项指标——触发强度、移植效应、截断后安慰剂、可恢复性、行为有效截断点、回忆准确性分离度——并设置数据依赖的显式判定门控。在15个来自7家厂商的模型上,针对258节点的时间校准宏观面板进行测试,揭示三种模式:(i) 日期触发反射并非规模现象,而是追踪训练近期性,具体成因未明:所有可测量的2024年开源权重模型均未显现,包括一个700亿参数级且截断对齐回忆倾向的模型;而所有测试的2026年生成模型均表现出该特征,并在同一家族的MoE架构中(Qwen3 → Qwen3.6)于约30亿活跃参数处突然出现;(ii) 有效截断点跨度达22个月,早于厂商报告日期最多8个月,否定日历窗口安慰剂的有效性;(iii) 结果非服务不变:以BF16服务一个原为FP8参考的模型会破坏触发估计的稳定性,而AWQ-INT4则保持稳定;提供方锁定的推理机制使某一探针无法收敛。因此,协议将量化方式与思维范式明确纳入其合同范围。我们公开发布面板数据、预注册、审计行、对话记录及一键重生成脚本。
原文摘要 · Abstract (English)
Large language models leak parametric knowledge of what followed a historical date into decision tasks indexed by that date -- not necessarily a lookup of the realized outcome, but knowledge of the period all the same. Existence is settled; what users lack is a cheap way to audit a given model. We present HindsightBench, a black-box audit protocol that profiles parametric hindsight in any time-indexed LLM decision task at probe-level cost (no backtests, no logprobs, no corpus access). It chains a four-arm date-manipulation matrix (revealed/date-only/masked/transplanted), dual memory probes (date recovery; outcome recall), and six metrics -- trigger strength, transplant effect, post-cutoff placebo, recoverability, behaviorally effective cutoff, and recall-accuracy dissociation -- with explicit gates where identifiability is data-dependent. Applied to 15 models from seven vendors on a 258-node vintage-correct macro panel, it yields three patterns: (i) the date-trigger reflex is not a scale phenomenon -- it tracks training recency, though what installs it is not identified here: absent across every 2024 open-weight row where it is measurable, including a 70B tier with cutoff-aligned recall propensity, present in every tested 2026-generation model, and switching on within one vendor lineage (Qwen3 -> Qwen3.6) in the same MoE family at ~3B active; (ii) effective cutoffs span 22 months across vendors and precede vendor-reported dates by up to eight months, invalidating calendar-window placebos; (iii) results are not invariant to serving -- BF16 serving of an FP8-referenced model breaks the trigger estimate's stability while AWQ-INT4 preserves it, and a provider-locked reasoning regime makes one probe non-convergent -- so the protocol pins quantization and thinking regime as part of its contract. We release the panel, preregistrations, audit rows, transcripts, and one-command regeneration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。