让语言模型自动总结经验,持续自我改进。
Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

- 通过在线提取模型轨迹中的经验模式,构建可复用的程序化记忆
- 在SCIKNOWEVAL和LIVECODEBENCH上比SDPO提升7.9%-13.6%
- 适合追求自进化能力的语言模型研究者
基于可验证奖励的强化学习(RLVR)及其自蒸馏变体(如SDPO)通过验证器评估每轮推演并根据回合级信号更新策略。然而,推演中蕴含的丰富程序信息通常未被保留或重用。在不同回合与训练周期中,模型反复遇到相关问题且策略不断变化,产生回合间信号——哪些策略始终通过验证、哪些失败模式持续存在、哪些模式反复出现。本文提出程序化记忆蒸馏(PMD),将这些跨回合信号转化为可复用的程序化记忆,并在训练中将其蒸馏进策略权重。该记忆作为训练支架,最终融入策略自身,实现推理时无记忆。PMD在三个抽象层次组织记忆:原始轨迹、自我反思的策略与教训、跨问题重复的行为模式,均从模型自身轨迹中在线提取。一个记忆条件化的自教师利用累积经验监督学生模型的推演,使学生逐步将程序知识内化到参数中。核心设计原则为共演化:策略生成推演以更新记忆,记忆又塑造监督信号以更新策略。实证结果表明,在Qwen3-8B和OLMo3-Instruct-7B上,PMD在SCIKNOWEVAL上较SDPO提升3.8%-5.5%,在LIVECODEBENCH上提升7.9%-13.6%。共演化机制驱动性能提升:冻结记忆或策略时,性能下降超10%。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。