不训练即可加速视频生成,通过智能压缩缓存提升效率。
PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache
- 动态压缩键值缓存,保留关键语义和时空结构。
- 48帧视频生成提速1.7至2.2倍,末段帧提速达3.7倍。
- 适合需要长视频生成且注重推理效率的研究者。
统一自回归模型将多模态任务(如文本、图像、视频)统一为共享词元空间下的序列建模问题。此类模型依赖键值缓存(KV-cache)将注意力计算从O(T²)降低至O(T),但缓存大小随生成词元数量线性增长,成为制约推理效率和生成长度的主要瓶颈。统一自回归视频生成继承此问题。分析发现,缓存词元具有显著的时空特性:(i) 文本与条件图像词元作为持久语义锚点,持续获得高注意力;(ii) 对前序帧的注意力随时间距离自然衰减。基于此,我们提出PackCache,一种无需训练的缓存管理方法,通过三种协同机制实现动态压缩:条件锚定保留语义参考,跨帧衰减建模按时间距离分配缓存预算,空间保持位置嵌入维持删除后的3D结构一致性。在48帧长序列上,端到端生成速度提升1.7–2.2倍;尤其在最后一段四帧(受缓存膨胀影响最严重、成本最高),在A40和H200上分别提速2.6倍和3.7倍,展现出实现长序列视频生成的强大潜力。
原文摘要 · Abstract (English)
A unified autoregressive model is a Transformer-based framework that addresses diverse multimodal tasks (e.g., text, image, video) as a single sequence modeling problem under a shared token space. Such models rely on the KV-cache mechanism to reduce attention computation from O(T^2) to O(T); however, KV-cache size grows linearly with the number of generated tokens, and it rapidly becomes the dominant bottleneck limiting inference efficiency and generative length. Unified autoregressive video generation inherits this limitation. Our analysis reveals that KV-cache tokens exhibit distinct spatiotemporal properties: (i) text and conditioning-image tokens act as persistent semantic anchors that consistently receive high attention, and (ii) attention to previous frames naturally decays with temporal distance. Leveraging these observations, we introduce PackCache, a training-free KV-cache management method that dynamically compacts the KV cache through three coordinated mechanisms: condition anchoring that preserves semantic references, cross-frame decay modeling that allocates cache budget according to temporal distance, and spatially preserving position embedding that maintains coherent 3D structure under cache removal. In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences, showcasing its strong potential for enabling longer-sequence video generation. Notably, the final four frames - the portion most impacted by the progressively expanding KV-cache and thus the most expensive segment of the clip - PackCache delivers a 2.6x and 3.7x acceleration on A40 and H200, respectively, for 48-frame videos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。