arXiv:2606.23459cs.CL2026-06

测试大模型在无提示下主动记住并执行隐含任务的能力

TriggerBench: Investigating Prospective Memory for Large Language Models

论文配图:TriggerBench: Investigating Prospective Memory for Large Language Models
图 1 · 摘自论文原文
  • 设计多维度对比实验,评估模型自发记忆与响应能力
  • 长上下文下主动记忆准确率显著下降,比回顾记忆难得多
  • 能反映模型剩余推理能力,适合研究智能体的内在资源

大型语言模型在长时间交互中日益普及,但现有评估主要聚焦于通过明确提问的回溯记忆(RM)。主动记忆(PM)——即在无直接提示下自发回忆并执行潜在约束的能力——尚未被充分评估。我们提出TriggerBench,一个涵盖日常助手与专业工作流的五维主动记忆基准。该基准通过匹配的回溯记忆对照、对比性正负样本及过载触发器,实现对主动回忆、误报率和注意力鲁棒性的统一量化。评估发现:(i) PM存在精度-召回权衡与注意力脆弱性;增强推理虽提升主动回忆,但模型易陷入‘始终提醒’的过拟合策略;在隐式约束或请求过载时,PM准确率大幅下降。(ii) PM明显难于RM:相同情境下,RM在10万词符内近乎饱和,而PM随上下文长度迅速衰减。(iii) PM可作为推理余量的行为探针:将PM任务与AIME-2025数学题结合,发现成功轨迹的PM准确率高于失败轨迹,说明其捕捉了令牌数无法体现的剩余推理能力。

原文摘要 · Abstract (English)

While Large Language Models (LLMs) are increasingly deployed in long interactions, existing evaluations focus predominantly on retrospective memory (RM) via explicit queries. Prospective memory (PM), the critical ability to spontaneously recall and act on latent constraints without direct prompts, remains largely unevaluated. We introduce TriggerBench, a comprehensive PM benchmark spanning five dimensions across both daily assistants and professional workflows. TriggerBench pairs scenarios with matched RM controls, contrastive positive/negative variants, and overloaded triggers, enabling fine-grained measurement of proactive recall, false-alarm rate, and attentional robustness under a single protocol. Our evaluation yields three key findings. (i) PM shows a precision-recall trade-off and attentional fragility. Though enhanced reasoning significantly improves proactive recall, models may overfit to an "always-remind" heuristic. Furthermore, PM accuracy degrades substantially under implicit constraints or triggers overloaded by concurrent user requests, indicating that robust PM remains an open challenge. (ii) PM is notably harder than RM: on identical contexts, RM near-saturates up to 100K tokens, while PM decays sharply as context length scales. (iii) PM may serve as a behavioral probe of spare reasoning capacity. Pairing PM scenarios with AIME-2025 math problems reveals that successful trajectories yield higher PM accuracy than failed ones at the same context length, showing PM tracks spare reasoning budget that token count obscures. Project page: https://github.com/KristenZHANG/TriggerBench-Official.

主动记忆大模型评估推理能力智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。