提出新方法让稀疏编码保留关键解码信号，避免信息丢失。
Decoder-Preserving Sparse Autoencoders: Which Readouts Survive Sparse Compression?
- 用解码器保真损失+重构误差联合训练，确保重要信号不被压缩丢弃。
- 在GPT-2小模型上降低10.6%-11.4%的解码失真，重构质量不变。
- 适合关注模型内部表征稳定性的研究者，尤其关注解码能力保持。
稀疏自编码器（SAEs）将模型激活压缩为稀疏代码，但相同重构误差和稀疏度下可能保留不同的线性可解码信号。本文将此歧义形式化为最优岭预测算子间的矩阵失真，并通过结合该失真与重构损失来训练解码器保真型SAE。在秩松弛假设下，各向同性任务先验在不改变主成分排序的情况下饱和每模式遗忘成本；而结构化先验则能改变保留模式。受控稀疏实验表明，声明的先验可保护其任务子空间外的组合。在GPT-2 small block 8上，DPSAE在三次配对运行中使持有样本解码失真降低10.6–11.4%，同时保持重构NMSE一致。相同检查点通过平均自然文本输出KL非劣性测试，但其中一个与Pythia配对的模型在仅依赖少数稀疏特征的探测任务中未见提升。结果表明，重构质量并不决定哪些重拟合线性读出能存活于稀疏压缩中，且读出保真独立于学习更清晰基准概念或保留所有冻结模型行为。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) compress model activations into sparse codes, but equal reconstruction error and sparsity can preserve different linearly decodable signals. We formalize this ambiguity as a matrix-valued distortion between optimal ridge-prediction operators and train decoder-preserving SAEs by combining this distortion with reconstruction loss. In a rank relaxation, an isotropic task prior saturates per-mode omission costs without changing PCA's ordering, whereas a structured prior can change which modes are retained. A controlled sparse experiment shows that a declared prior protects held-out combinations from its task subspace. On GPT-2 small block 8, DPSAE reduces held-out decoder distortion by 10.6--11.4% across three paired runs while matching reconstruction NMSE. The same checkpoints pass an average natural-text output-KL noninferiority test, but one matched Pythia pair shows no improvement in probes restricted to a few sparse features. These results show that reconstruction quality does not determine which refitted linear readouts survive sparse compression, and that readout preservation is distinct from learning cleaner benchmark concepts or preserving every frozen-model behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。