通过感知激活与影响的秩选择，实现大模型高效压缩。
Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
- 基于SVD的压缩框架，用反向信号影响引导每层权重矩阵的低秩近似。
- 在保留≤60%参数时，困惑度提升超18%，用90%更少校准数据达到相同效果。
- 可与端到端方法结合，显著降低计算量、内存和单次生成延迟。
我们提出激活与影响感知秩（AIR），一种基于SVD的大语言模型压缩框架，通过反向信号影响度量指导每个权重矩阵的低秩近似。从SVD-LLM(W)的激活感知最优解出发，AIR仅需一次闭式交替最小二乘（ALS）迭代，在单调下降保证下逐元素整合影响信息。AIR具有层内局部性，且可与端到端方法正交组合：单独使用即超越ACIP，结合LoRA后性能进一步提升。AIR在参数保留率≤60%时，困惑度较SVD-LLM(W)提升超过18%；仅需约10%的校准数据即可达到其质量水平；并将参数节省转化为计算量（FLOP）、峰值内存及单标记延迟的显著降低。
原文摘要 · Abstract (English)
We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric. Starting from the activation-aware optimum of SVD-LLM(W), AIR runs a single closed-form alternating least squares (ALS) sweep that integrates influence element-wise under a monotone-descent guarantee. AIR is layer-local and composes orthogonally with end-to-end methods: alone it exceeds ACIP, and AIR+LoRA outperforms it further. AIR improves perplexity over SVD-LLM(W) by >18% at <=60% parameter retention, matches its quality with ~90% less calibration data, and turns parameter savings into FLOP, peak-memory, and per-token latency gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。