用少量缓存线实现高效图像自回归生成,大幅节省内存并提速。
Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
- 按行压缩视觉缓存,保留关键依赖区域,淘汰低信息量内容。
- 仅用1/6至1/8缓存,图像质量反而提升,FID降低0.09至1.00。
- 无需训练,适配多模型,适合部署资源受限的生成系统。
自回归视觉生成因可扩展性和通用性成为图像与多模态合成的重要范式,但其解码过程需缓存所有已生成视觉标记,导致严重内存瓶颈和低吞吐。本文提出训练无关的逐级键值(KV)缓存压缩方案LineAR,利用视觉注意力的内在特性,在二维视角下按行管理缓存,通过行间注意力引导,保留视觉依赖区域,逐步剔除对后续行生成无害的低信息量标记。LineAR仅需极少缓存即可实现高效自回归图像生成,同时显著降低内存占用、提升速度,且保持或提升生成质量。在六种自回归图像生成模型上验证了其有效性与通用性:在LlamaGen-XL和Janus-Pro-1B上,ImageNet FID从2.77降至2.68,COCO FID从23.85降至22.86,仅保留1/6 KV缓存;Lumina-mGPT-768的DPG也获提升,仅用1/8缓存。此外,内存减少达67.61%,速度提升7.57倍(LlamaGen-XL),Janus-Pro-7B内存减少39.66%,速度提升5.62倍。
原文摘要 · Abstract (English)
Autoregressive (AR) visual generation has emerged as a powerful paradigm for image and multimodal synthesis, owing to its scalability and generality. However, existing AR image generation suffers from severe memory bottlenecks due to the need to cache all previously generated visual tokens during decoding, leading to both high storage requirements and low throughput. In this paper, we introduce \textbf{LineAR}, a novel, training-free progressive key-value (KV) cache compression pipeline for autoregressive image generation. By fully exploiting the intrinsic characteristics of visual attention, LineAR manages the cache at the line level using a 2D view, preserving the visual dependency regions while progressively evicting less-informative tokens that are harmless for subsequent line generation, guided by inter-line attention. LineAR enables efficient autoregressive (AR) image generation by utilizing only a few lines of cache, achieving both memory savings and throughput speedup, while maintaining or even improving generation quality. Extensive experiments across six autoregressive image generation models, including class-conditional and text-to-image generation, validate its effectiveness and generality. LineAR improves ImageNet FID from 2.77 to 2.68 and COCO FID from 23.85 to 22.86 on LlamaGen-XL and Janus-Pro-1B, while retaining only 1/6 KV cache. It also improves DPG on Lumina-mGPT-768 with just 1/8 KV cache. Additionally, LineAR achieves significant memory and throughput gains, including up to 67.61% memory reduction and 7.57x speedup on LlamaGen-XL, and 39.66% memory reduction and 5.62x speedup on Janus-Pro-7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。