提出记忆机制让模型有选择地保留历史信息,提升长文本理解能力。
MMLA: How Memory Lets the Past Shape the Future
- 设计有限记忆模块,动态决定哪些历史信息值得保留
- 在多跳问答任务中比基线模型提升5.5至16.6的F1值
- 适合需要长期依赖建模的复杂推理场景
长上下文可回放历史,但无法判断哪些已完成的观察值得信赖。MMLA在瞬时上下文与缓慢权重更新之间构建了有界驻留记忆。一个完成的局部段落被事件化;对每个事件,目标条件构造器提出语义内容,可信组装器生成完整版本行;部署时要么原子提交该行,要么返回空。训练时可为未来行为定价,部署时保持因果性和未来盲性。受控实验验证组件有效性:三个种子下300/300测试记录实现精确执行。带全档案回退的校准选择在多跳问答上优于弱预算匹配密集基线5.5–16.6 F1,优于BM25 4.0–6.2 F1;原版Llama预算执行失败,修正后的Qwen打包器满足每条记录容量限制。类型锚点-填充传输达到每种子240/240精确度,而三个同检查点对照组仅获0/240整体成功。当前障碍:集成循环未完成。基于V28(冻结的352.3M参数模型,由50.0M token原生支架试点生成)训练的密集行、结构跨度和检查点原生阅读器,在三个种子下均未能通过语义资格测试。候选精确度为0–45/23,040,查询精确度为10–1,536/9,216,记录宏观Brier值接近1,025类均匀参考水平。结构映射通过,但九个任务均未达标。因此预测性覆盖被关闭。
原文摘要 · Abstract (English)
Proposal. Long context can replay history, but it does not decide which completed observations deserve authority. MMLA formalizes a bounded resident memory between transient context and slow weight updates. A completed local segment is eventized; for each event, a target-conditioned constructor proposes semantic content and a trusted assembler produces a complete versioned row; deployment either commits that row atomically or returns NULL. Realized futures may price actions during training, while deployment remains causal and future-blind. Validated components. Controlled studies establish narrower ingredients. Lifecycle execution is exact on 300/300 held-out records for each of three seeds. Calibrated selection with full-archive fallback improves over a weak budget-matched dense baseline by 5.5--16.6 F1 and over BM25 by 4.0--6.2 F1 on held-out multi-hop QA; the original Llama budget execution is retained as failed, while the corrected Qwen packer satisfies the stated per-record caps. Typed anchor--filler transport reaches 240/240 held-out exactness per seed while three same-checkpoint controls obtain 0/240 whole-record successes. Current blocker. The integration loop is not complete. Dense-row, structured-span, and checkpoint-native readers trained from V28, the frozen 352.3M-parameter model-only checkpoint produced by a 50.0M-token native-scaffold pilot, all fail semantic qualification across three seeds. Candidate exactness is 0--45/23,040, query exactness is 10--1,536/9,216, and record-macro Brier remains near the 1,025-class uniform reference. Structural mapping passes, but none of the nine jobs qualifies. Predictive overwrite is therefore closed by gate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。