发现线性注意力大模型状态秩的分层现象，揭示低秩头对推理至关重要。
State Rank Dynamics in Linear Attention LLMs
- 识别出注意力头在运行时呈现低秩与高秩两种稳定状态
- 低秩头对模型推理不可或缺，高秩头存在显著冗余
- 提出零样本秩归一化剪枝，减少38.9%缓存开销
线性注意力大语言模型（Linear Attention LLMs）通过固定大小的状态矩阵压缩上下文，实现常数时间推理，但其内部状态动态机制仍不清晰。本文全面研究了先进线性注意力模型的运行时状态动态，发现一种称为‘状态秩分层’的根本现象：部分注意力头的有效秩持续接近零，另一部分则快速增长并收敛至上限。跨多种推理场景的实验表明，该动态高度一致，说明头的低秩或高秩属性是预训练中形成的内在结构特征，而非依赖输入的瞬态状态。诊断分析进一步揭示功能差异：低秩头对模型推理至关重要，而高秩头具有显著冗余。基于此，我们提出零样本秩归一化剪枝（Joint Rank-Norm Pruning），在几乎不损失精度的前提下，实现KV缓存开销下降38.9%。
原文摘要 · Abstract (English)
Linear Attention Large Language Models (LLMs) offer a compelling recurrent formulation that compresses context into a fixed-size state matrix, enabling constant-time inference. However, the internal dynamics of this compressed state remain largely opaque. In this work, we present a comprehensive study on the runtime state dynamics of state-of-the-art Linear Attention models. We uncover a fundamental phenomenon termed State Rank Stratification, characterized by a distinct spectral bifurcation among linear attention heads: while one group maintains an effective rank oscillating near zero, the other exhibits rapid growth that converges to an upper bound. Extensive experiments across diverse inference contexts reveal that these dynamics remain strikingly consistent, indicating that the identity of a head,whether low-rank or high-rank,is an intrinsic structural property acquired during pre-training, rather than a transient state dependent on the input data. Furthermore, our diagnostic probes reveal a surprising functional divergence: low-rank heads are indispensable for model reasoning, whereas high-rank heads exhibit significant redundancy. Leveraging this insight, we propose Joint Rank-Norm Pruning, a zero-shot strategy that achieves a 38.9\% reduction in KV-cache overhead while largely maintaining model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。