用潜在差异度量优化视觉自回归生成的冗余剪枝,提速近2.4倍
Where to Refine, When to Stop: Rethinking Redundancy via Latent Discrepancy for Efficient Visual Autoregressive Generation

- 以潜在空间变化量化令牌贡献,更准识别冗余计算
- 在Infinity-8B上实现最高2.35倍推理加速,画质几乎不变
- 无需训练,适配不同提示词,动态跳过无用分支
视觉自回归(VAR)模型生成图像质量高,但在高分辨率下推理延迟严重。现有加速方法多依赖层特征的启发式剪枝,对复杂语义敏感,难以准确识别冗余计算且适应性差。本文从像素空间生成影响重新思考冗余问题,提出潜变量差异(Latent Discrepancy)作为统一度量,通过追踪生成过程中模型状态变化来量化令牌贡献。分析发现,基于图像潜在或像素空间信号能更准确识别冗余。进一步观察到,在无分类器引导(CFG)中,条件与非条件分支间的差异收敛趋势随提示词变化具有高动态性。据此提出无需训练的LD-Pruning框架,结合无解码区域选择与自适应非条件分支跳过机制,有效去除冗余。大量实验表明,该方法显著降低推理延迟,同时保持高质量生成,在Infinity-8B上实现最高2.35倍加速。
原文摘要 · Abstract (English)
Visual Autoregressive (VAR) models deliver high-quality image generation but suffer from significant inference latency at high resolutions. Recent acceleration approaches most rely on heuristic measures with layer features to prune tokens. Such heuristics are sensitive to complex contextual semantics, leading to inaccurate identification of redundant computation and poor adaptability across prompts. We rethink redundancy in VAR from the perspective of its impact on pixel-space generation and introduce Latent Discrepancy. This unified metric quantifies a token's contribution by measuring the change in model states during generation. Our analysis shows that redundancy is more accurately identified when guided by image latent or pixel-space signals. We further observed that in classifier-free guidance (CFG), the convergence trend of the discrepancy between conditional and unconditional branches exhibits high dynamics with different prompts. Based on these findings, we propose LD-Pruning (Latent Discrepancy Pruning), a training-free framework that removes redundancy via latent discrepancy by integrating decoding-free region selection and adaptive unconditional-branch skipping. Extensive experiments show that LD-Pruning substantially reduces inference latency while maintaining high generation quality, achieving up to 2.35x speedup on Infinity-8B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。