用原始注意力日志提升大模型解释力与压缩效率
RCStat: A Statistical Framework for using Relative Contextualization in Transformers
- 基于注意力原始日志构建相对上下文度量，避免软最大值失真
- 实现键值缓存压缩40%以上，质量损失低于5%；解释精度显著优于传统方法
- 无需重训练即可用于模型解释和内存优化，适合部署场景
自回归变换器中输入词元重要性分析通常依赖软最大值归一化的注意力权重，这掩盖了预软最大值查询-键值日志的丰富结构。我们提出RCStat，一个基于相对上下文化（RC）的统计框架，通过测量词元片段间的上下文对齐性来利用原始注意力日志，并推导出RC的高效上界。该框架应用于两项任务：(i) 键值压缩，基于RC设定阈值实现自适应键值淘汰，大幅减少缓存占用且质量损失极小；(ii) 属性分析，相比后软最大值方法，提供更精确的词元、句子及块级解释。在问答、摘要与属性分析基准测试中，RCStat均取得显著性能提升，达到当前最优压缩与解释效果，且无需模型重训练。
原文摘要 · Abstract (English)
Prior work on input-token importance in auto-regressive transformers has relied on Softmax-normalized attention weights, which obscure the richer structure of pre-Softmax query-key logits. We introduce RCStat, a statistical framework that harnesses raw attention logits via Relative Contextualization (RC), a random variable measuring contextual alignment between token segments, and derive an efficient upper bound for RC. We demonstrate two applications: (i) Key-Value compression, where RC-based thresholds drive adaptive key-value eviction for substantial cache reduction with minimal quality loss; and (ii) Attribution, where RC yields higher-fidelity token-, sentence-, and chunk-level explanations than post-Softmax methods. Across question answering, summarization, and attribution benchmarks, RCStat achieves significant empirical gains, delivering state-of-the-art compression and attribution performance without any model retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。