用稀疏编码解析大模型键值缓存的语义结构，让内存瓶颈变可解释。
Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoders
- 用顶K稀疏自编码器分解键值缓存为可读的语义单元。
- 发现键向量稀疏而值向量密集，需不同预算策略。
- 保留关键语义信息，性能接近原模型，适合可解释性研究。
键值（KV）缓存是长上下文大语言模型的主要内存瓶颈，但通常被视为不可见的数值张量。本文提出STA-Attention框架，利用顶K稀疏自编码器（SAEs）将KV缓存分解为可解释的“语义原子”。与标准$ L_1 $正则化SAE不同，顶K方法消除了压缩偏差，保持了注意力所需的精确点积几何关系。分析揭示了根本性的“键值不对称”：键向量作为高度稀疏的路由，由“语义拐点”主导；深层值向量承载密集内容，需更大预算。基于此结构，我们提出双预算策略，选择性保留最信息丰富的语义成分，同时过滤表示噪声。在Yi-6B、Mistral-7B、Qwen2.5-32B等模型上的实验表明，我们的语义重构在困惑度和零样本性能上与原模型相当，有效弥合了机制可解释性与忠实注意力建模之间的差距。
原文摘要 · Abstract (English)
The Key-Value (KV) cache is the primary memory bottleneck in long-context Large Language Models, yet it is typically treated as an opaque numerical tensor. In this work, we propose \textbf{STA-Attention}, a framework that utilizes Top-K Sparse Autoencoders (SAEs) to decompose the KV cache into interpretable ``semantic atoms.'' Unlike standard $L_1$-regularized SAEs, our Top-K approach eliminates shrinkage bias, preserving the precise dot-product geometry required for attention. Our analysis uncovers a fundamental \textbf{Key-Value Asymmetry}: while Key vectors serve as highly sparse routers dominated by a ``Semantic Elbow,'' deep Value vectors carry dense content payloads requiring a larger budget. Based on this structure, we introduce a Dual-Budget Strategy that selectively preserves the most informative semantic components while filtering representational noise. Experiments on Yi-6B, Mistral-7B, Qwen2.5-32B, and others show that our semantic reconstructions maintain perplexity and zero-shot performance comparable to the original models, effectively bridging the gap between mechanistic interpretability and faithful attention modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。