arXiv:2511.21730cs.CLcs.AI2025-11被引 2

首个评估语言智能体程序记忆检索的基准,揭示现有方法在新任务中的严重退化问题。

A Benchmark for Procedural Memory Retrieval in Language Agents

  • 构建双语料库分离程序检索与执行,测试跨对象实例的等效流程识别能力
  • 嵌入方法在熟悉场景表现好,但在新场景中性能骤降,而LLM生成的抽象更稳定
  • 发现当前编码器存在架构瓶颈,大规模语料比改进表示更有效

当前AI智能体在熟悉环境中表现优异,但在面对词汇未见的新任务时却迅速失效——这是程序记忆系统的核心局限。我们提出了首个将程序记忆检索与任务执行分离的基准,评估智能体能否识别跨越不同对象实例的功能等效流程。基于ALFWorld构建专家与LLM生成轨迹的双语料库,对六种检索方法进行系统分层查询评估。结果揭示明显的泛化断崖:嵌入方法在熟悉情境中表现强劲,但在新情境中显著退化;而LLM生成的程序抽象展现出可靠的跨上下文迁移能力。控制消融实验表明,尽管嵌入能捕捉部分词法抽象,但本质上将流程视为无序词袋,丢失了跨上下文迁移所必需的时间结构。语料规模带来的增益远超表示增强,揭示了当前编码器的架构天花板。该基准提供首个诊断框架,区分真正的程序理解与表面记忆,并为开发具备可靠泛化能力的检索系统提供工具。资源已开源于GitHub(https://github.com/qpiai/Proced_mem_bench)。

原文摘要 · Abstract (English)

Current AI agents excel in familiar settings, but fail sharply when faced with novel tasks with unseen vocabularies -- a core limitation of procedural memory systems. We present the first benchmark that isolates procedural memory retrieval from task execution, evaluating whether agents can recognize functionally equivalent procedures that span different object instantiations. Using ALFWorld, we construct dual corpora of expert and LLM-generated trajectories and evaluate six retrieval methods using systematically stratified queries. Our results expose a clear generalization cliff: embedding-based methods perform strongly on familiar contexts, yet degrade considerably on novel ones, while LLM-generated procedural abstractions demonstrate reliable cross-context transfer. Controlled ablations show that although embeddings capture some lexical-level abstraction, they fundamentally treat procedures as unordered bags of words, discarding temporal structure necessary for cross-context transfer. Corpus scale delivers far larger gains than representation enrichment, revealing an architectural ceiling in current encoders. Our benchmark offers the first diagnostic framework separating genuine procedural understanding from surface-level memorization and gives tools for developing retrieval systems capable of dependable generalization. Resources available at our GitHub repository (https://github.com/qpiai/Proced_mem_bench).

程序记忆检索基准语言智能体泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。