用输出空间误差指导模型压缩，能提升准确率但可能牺牲流畅性。
Output-Space Allocation Costs for Calibration-Guided LLM Compression: An Empirical Study
- 以输出空间重建误差作为压缩成本，替代传统权重空间误差。
- 在Qwen3-8B上压缩50%时，准确率提升0.8个百分点，但困惑度上升16%。
- 适用于追求高精度的场景，尤其在中高压缩比下效果更明显。
无需训练的大型语言模型压缩方法常利用校准数据指导压缩决策。ROCKET通过稀疏字典分解结合多选背包问题（MCKP）进行层内因子分解，其因子分解基于输出重建目标，但采用权重空间的Frobenius误差作为MCKP分配成本。本文探究将分配成本与输出空间目标对齐是否能提升压缩模型的保真度。在Qwen3-8B模型上以50%压缩率测试，所提方法ROCKET-ActCost在8个零样本基准上平均准确率提升0.8个百分点（53.1% vs 52.3%），但WikiText困惑度上升16%（61.46 vs 52.98）。该准确率与困惑度的权衡表明，不同分配目标偏好不同下游指标。权重空间与输出空间误差相关性超过0.99，限制了分配偏差，解释了效果提升有限的原因。在Llama-3.2-1B模型上20%压缩率下，两种方法结果几乎相同（准确率53.3% vs 53.5%，困惑度14.45 vs 14.66），表明成本函数影响在低压缩比时较小。
原文摘要 · Abstract (English)
Training-free compression methods for large language models (LLMs) often use calibration data to guide compression decisions. ROCKET, a recent method combining sparse-dictionary factorization with multi-choice knapsack problem (MCKP) allocation, derives its per-layer factorization from an output reconstruction objective but uses weight-space Frobenius error as the MCKP allocation cost. We investigate whether aligning the allocation cost with the output-space objective improves compressed model fidelity. On Qwen3-8B at 50\% compression, our ROCKET-ActCost achieves +0.8 percentage points higher average accuracy across 8 zero-shot benchmarks (53.1\% vs 52.3\%), but increases WikiText perplexity by 16\% (61.46 vs 52.98). This accuracy-perplexity tradeoff reveals that different allocation objectives favor different downstream metrics. The high correlation ($>$0.99) between weight-space and output-space errors limits allocation divergence, explaining the modest effect size. On Llama-3.2-1B at 20\% compression, the two methods produce near-identical results (53.3\% vs 53.5\% accuracy, 14.45 vs 14.66 PPL), suggesting that the effect of the cost function is minor at lower compression ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。