提出HACK++压缩方法,让视觉自回归模型生成更快更省内存。
HACK++: Towards More Effective Head-Aware Key-Value Compression for Efficient Visual Autoregressive Modeling

- 按注意力头功能分两类,分别设计压缩策略。
- 仅用30%注意力预算和10%缓存预算,保持几乎无损生成质量。
- 无需训练,适配多种图像生成任务,缓存压缩可低至1%。
视觉自回归(VAR)模型采用逐尺度预测范式,以较少解码步数实现高质量生成。然而,现有模型因多尺度键值(KV)缓存累积导致显著的注意力复杂度和严重内存开销。本文将缓存压缩引入逐尺度范式,通过深入分析发现:注意力头可分为两类——语义一致性保持的上下文头与空间连贯性保持的结构化头。二者功能差异明显,且对历史尺度的依赖随层和生成步骤变化,需动态分配缓存预算。为此,提出HACK++:一种无需训练的头感知压缩框架。通过一次离线校准,分类头类型并获取专属先验;推理时在独立预算下解耦注意力与缓存压缩,约束当前尺度注意力成本,同时以模式特异性策略和依赖感知预算,大幅压缩累积缓存。在多个文本到图像、类别条件及统一理解-生成任务上的实验验证其有效性与泛化性。例如,在Infinity-2B/8B上,仅需30%注意力预算和10%缓存预算即可近似无损生成,即使缓存预算降至1%仍具鲁棒性。
原文摘要 · Abstract (English)
Visual Autoregressive (VAR) models adopt a next-scale prediction paradigm, offering high-quality generation with substantially fewer decoding steps. However, existing VAR models suffer from significant attention complexity and severe memory overhead due to the accumulation of key-value (KV) caches across scales. In this paper, we tackle this challenge by introducing KV cache compression into the next-scale paradigm. We begin with an in-depth analysis of VAR attention and observe that attention heads can be stably divided into two functionally distinct categories: Contextual Heads focus on maintaining semantic consistency, while Structural Heads preserve spatial coherence. Their functional divergence makes existing one-size-fits-all compression methods perform poorly on VAR models. We further find that the two head types differ markedly in their reliance on historical scales, and that this reliance shifts across layers and generation steps, arguing for an adaptive cache budget allocation. To address these challenges, we propose HACK++, a training-free Head-Aware key-value Compression frameworK for VAR models. From a one-time offline calibration, HACK++ classifies head types and derives head-specific priors. At inference, it decouples attention from cache compression under independent budgets, bounding the current-scale attention cost while compressing the accumulated cache far more aggressively, via pattern-specific strategies and a reliance-aware budget allocation. Extensive experiments on multiple VAR models across text-to-image, class-conditional, and unified understanding-and-generation tasks validate the effectiveness and generalizability of HACK++. For example, on Infinity-2B/8B, HACK++ maintains near-lossless generation with only a 30% attention budget and a 10% cache budget, and remains robust even under a 1% cache budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。