arXiv:2607.16254cs.LG2026-07

发现大模型FFN层的写入有正负之分,影响长文本检索结果。

More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrieval

论文配图:More Than Memory: Task-Conditioned Signed FFN Writes in Long-Context Retrieval
图 1 · 摘自论文原文
  • 通过逐层缩放FFN写入,发现其方向性受任务调控
  • 7/8场景中最终FFN层为抑制器,60%层在不同任务间角色切换
  • 写入梯度可作为判断抑制/增强作用的紧凑诊断指标

FFN常被视为参数化记忆。但在长上下文检索中,关键问题不仅是其存储内容,更在于其原始残差写入是否推动当前检索状态趋向或偏离正确答案。本文通过逐层缩放模型自身FFN写入(不修改权重或注入外部向量),在控制条件下测试了这一现象。在多个字面与语义检索任务中,原始FFN响应曲面具有符号性、层特定性和任务依赖性:在8个模型套件中的7个场景里,最后一层FFN为抑制器;60%的层在不同检索模式间切换角色(95%置信区间[50%, 69%])。沿原生写入方向的局部方向导数可区分两种单调角色:34/35例中抑制器导数为负,18/18例中放大器导数为正,表明角色不可简化为写入大小。在经安全过滤的LongBench检索问答探针上,该诊断对原始数据的预测相关性达R²=0.796(Qwen2.5-7B)和0.791(Qwen3.5-9B);保留抑制器-衰减策略的模型在检索边界上优于随机与范数匹配对照组。结果表明,原生FFN缩放揭示了检索中具有符号性与任务依赖性的残差写入结构,且写入梯度对齐是判断两种单调角色的紧凑诊断工具。

原文摘要 · Abstract (English)

FFNs are often treated as parametric memories. In long-context retrieval, however, the sharper question is not only what they store, but whether their native residual writes push the current retrieval state toward or away from the correct answer. We test this by scaling the model's own FFN write one layer at a time, without editing weights or injecting external steering vectors. Across controlled literal and semantic retrieval suites, native FFN response surfaces are signed, layer-specific, and task-conditioned: the final FFN is a suppressor in 7 of 8 model-suite cases, and 60% of layers switch role between retrieval modes (95% CI [50%, 69%]). A local directional derivative along the native write separates the two monotone roles: suppressors have negative derivative in 34/35 cases, and amplifiers have positive derivative in 18/18 cases, so the roles are not reducible to write size. On a safety-filtered LongBench retrieval-QA probe, the same diagnostic predicts attenuation damage with raw R^2=0.796 on Qwen2.5-7B and 0.791 on Qwen3.5-9B; a held-out suppressor-attenuation policy improves retrieval margins over random and norm-matched controls. These results show that native FFN scaling exposes a signed, task-conditioned residual-write structure in retrieval, and that write-gradient alignment is a compact diagnostic for the two monotone roles.

长文本检索FFN机制写入方向模型可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。