发现语言注意力可压缩，数据本身决定压缩潜力。
Compressible Softmax-Attended Language under Incompressible Attention
- 拆解注意力逻辑场为学习与生成两部分，分别分析谱特性。
- 90%注意力能量仅需2-11个奇异分量即可捕获，远低于头维度。
- 模型架构差异小，压缩性由数据决定，非模型框架导致。
Softmax注意力通过$ d_h $个头维度定义交互，但真实文本通过后各维度权重不均。本文将注意力逻辑场分解为学习成分与生成成分，并分别测量其谱特性。在五个Transformer语言模型（参数量124M–7B，四类架构）的5,888个KV头中，逻辑能量场$\tilde{E}$在2–11个奇异分量内即可承载90%方差。学习交互矩阵$W_Q^\mathrm{T} W_K$需38–75个分量才能达到相同阈值，$d_h \in \{64, 128\}$。有效秩的谱间隙达5–25倍。注意力的可压缩性是数据属性，而非分析框架所致。
原文摘要 · Abstract (English)
Softmax attention defines an interaction through $d_h$ head dimensions, but not all dimensions carry equal weight once real text passes through. We decompose the attention logit field into a learned component and a generated component and measure their spectra separately. For all 5,888 KV heads in five transformer language models (124M--7B parameters, four architecture families), the logit energy field $\tilde{E}$ reaches 90\% of its variance in 2--11 singular components. The learned interaction matrix $W_Q^\mathrm{T} W_K$ needs 38--75 components for the same threshold out of $d_h \in {64, 128}$. The spectral gap is 5--25$\times$ in effective rank. The compressibility of softmax-attended language is a property of the data, not the frame that analyzes it.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。