给冻结的Qwen模型加记忆，不改结构不调参，用小模块实现可逆记忆增强。
Prometheus Mind: Retrofitting Memory to Frozen Language Models
- 用无监督方向发现技术提取语义向量，无需标注数据
- 在132个案例中准确率94.4%，非正式输入下降至19.4%
- 模块可一键移除，适合需安全记忆的部署场景
为预训练语言模型添加记忆通常需要修改架构或权重。我们提出Prometheus Mind，通过11个模块化适配器（530MB，7%开销）为冻结的Qwen3-4B模型添加记忆功能，且可完全移除。系统解决四大问题：(1) 提取——提出对比方向发现（CDD），通过最小对无监督识别语义方向；(2) 训练——端到端优化失败，采用分阶段训练各适配器于简单代理任务成功；(3) 注入——学习编码器泛化差，发现lm_head权重行已具备所需映射，无需训练；(4) 隐藏状态坍塌——变压器使'wife'与'brother'相似度超0.98，通过训练投影恢复区分性（0.98→0.09）。在PrometheusExtract-132数据集上，干净输入下检索准确率达94.4%（n=54，95%置信区间[84.9%, 98.1%]），非正式输入（省略、填充词、隐含主语）下降至19.4%（n=36）。主要瓶颈是关系分类（准确率47.3%），导致多数提取错误。
原文摘要 · Abstract (English)
Adding memory to pretrained language models typically requires architectural changes or weight modification. We present Prometheus Mind, which retrofits memory to a frozen Qwen3-4B using 11 modular adapters (530MB, 7% overhead) -- fully reversible by removing the adapters. Building this system required solving four problems: (1) Extraction -- we develop Contrastive Direction Discovery (CDD), which finds semantic directions via minimal pairs without labeled data. (2) Training -- end-to-end optimization collapses; stage-wise training of each adapter on simple proxy tasks succeeds. (3) Injection -- learned encoders fail to generalize; we find that lm_head-weight rows already provide the mapping we need, requiring no training. (4) Hidden state collapse -- transformers make ``wife'' and ``brother'' 0.98+ similar; we train projections to recover distinction (0.98 $\rightarrow$ 0.09). On PrometheusExtract-132 (132 cases), the system achieves 94.4% retrieval on clean inputs (n=54, 95% CI: [84.9%, 98.1%]), degrading to 19.4% on informal inputs with ellipsis, filler words, or implicit subjects (n=36). The primary bottleneck is relation classification (47.3% accuracy), responsible for most extraction errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。