用随机固定池塘让预训练模型跨推理步骤记住信息，无需额外训练。
The Reservoir Attention Network: Cross-Pass State in Pretrained Transformers via Content-Addressable Reservoir Injection
- 在注意力层注入随机初始化的动态池塘，实现跨前向传递的状态记忆。
- 在单张消费级显卡上成功测试了从124M到1.5B参数模型的跨步状态保持。
- 不需训练池塘，验证了无监督动态机制即可支撑持续智能行为。
本文对储层注意力网络（RAN）进行了可行性与动态性研究，该架构通过将一个固定且随机初始化的储层注入预训练Transformer的中间层注意力中，以实现跨前向传播的状态传递。实验覆盖GPT-2（124M、355M）至Qwen2.5（0.5B、1.5B）的多个模型，均在单张消费级GPU上完成。任务为最小探针，旨在隔离并检验单一机制；更广泛的持续活跃智能体愿景被视为计算受限的未来工作，非本文主张。储层设计为未训练（固定随机），以检验仅依赖未训练的循环动力学是否足以维持可用的跨步状态，而训练后的循环则作为更高成本的补充方向。
原文摘要 · Abstract (English)
A feasibility and dynamics study of the Reservoir Attention Network (RAN), an architecture that injects a fixed, randomly-initialized reservoir into the mid-layer attention of a pretrained transformer to carry state across forward passes. Experiments span GPT-2 (124M, 355M) to Qwen2.5 (0.5B, 1.5B) on a single consumer GPU. The tasks are minimal probes chosen to isolate individual mechanisms; the broader always-alive agent vision is treated throughout as compute-limited future work, not a claim of this paper. The reservoir is left untrained (fixed random) by design: this isolates whether untrained recurrent dynamics alone suffice to carry usable cross-pass state, leaving trained recurrence as a complementary, more expensive direction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。