用稀疏自编码器直接替换语言模型的缓存更新，实现精准控制。
WriteSAE: Sparse Autoencoders for Recurrent State
- 设计匹配模型写入格式的矩阵型稀疏自编码器
- 92.4%位置替换后输出分布更接近原模型，平均89.8%有效
- 首次在状态空间模型中实现缓存级干预，适合可控生成研究
我们提出WriteSAE，一种用于循环语言模型缓存写入矩阵的稀疏自编码器。在Gated DeltaNet、Mamba-2和RWKV-7中，每个词元向循环缓存写入一个矩阵更新；传统残差流SAE使用向量原子，无法直接替代。WriteSAE学习与模型写入同形状的秩一矩阵原子。测试时，在SAE激活位置，移除模型写入，代入缩放后的原子并继续前向传播。该替换在92.4%的评估位置上使最终词元分布更接近原模型，平均达89.8%。对Gated DeltaNet，基于遗忘门、读取查询和输出嵌入的公式可预测逻辑值变化，$R^2 = 0.98$。相同方法在Mamba-2-370M上达到88.1%效果。生成阶段，该公式可选择写入方向；将写入内容以三倍范数写入三个连续缓存位置，原本排名100–1000的词元在100%的续写中出现，较原始的33.3%显著提升。据我们所知，这是首个在状态空间或混合循环层中报告的缓存级操控干预。
原文摘要 · Abstract (English)
We introduce WriteSAE, a sparse autoencoder for the matrix updates written into recurrent language-model state. In Gated DeltaNet, Mamba-2, and RWKV-7, each token writes a matrix-shaped update to a recurrent cache; a residual-stream SAE has vector-shaped atoms and cannot replace that update directly. WriteSAE learns rank-1 matrix atoms with the same shape as the model's own write. This lets us test a direct replacement: at positions where the SAE activates an atom, we remove the model's write, insert the atom scaled by its SAE activation, and continue the forward pass. The atom gives a closer final token distribution than deleting the write on 92.4% of evaluated positions; averaged per atom, the rate is 89.8%. For Gated DeltaNet, a formula using the forget gate, read query, and output embedding predicts the resulting logit change with $R^2 = 0.98$. The same replacement test transfers to Mamba-2-370M at 88.1%. In generation, the formula chooses a write direction; writing it into three consecutive cache positions at $3\times$ the norm of the model's write makes tokens initially ranked 100--1000 by the unmodified model appear in 100% of continuations, up from 33.3%. To our knowledge this is the first cache-level steering intervention reported in a state-space or hybrid recurrent layer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。