通过参数化吸收技术,实现视频生成模型的高效内存压缩。
Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorption

- 将注意力层从全局关注转为局部关注,用参数调节补偿历史信息缺失。
- 在1.3B到14B模型上减少50%的键值缓存,视觉质量几乎无损。
- 适合需要长视频生成且受限于内存的实时推理场景。
自回归流式模型已成为长视频生成的强大范式。然而,线性增长的键值(KV)缓存成为主要瓶颈,导致内存溢出和推理吞吐下降。常见的压缩方法是丢弃冗余的KV标记,但这常破坏长程依赖,引发时间闪烁和身份丢失。本文提出实例特定参数吸收(ISPA),一种将KV缓存压缩从丢弃转向提炼的新框架。核心思想是通过将部分层从全注意力(F-Layers)转为内存高效的局部注意力(L-Layers),将历史上下文“吸收”进模型权重中。具体而言,在短暂预热阶段,ISPA监控全局与局部注意力之间的输出差异;在切换点,通过求解闭式最小二乘问题,计算出针对特定实例的权重调制,以补偿缺失的历史信息。跨多种架构(1.3B至14B)的实验表明,ISPA可在近乎无损视觉质量下移除高达50%的KV缓存。我们希望这一视角能推动未来研究探索超越外部标记级缓存管理的参数化记忆整合机制。
原文摘要 · Abstract (English)
Autoregressive (AR) streaming models have emerged as a powerful paradigm for long video generation. However, the linearly growing Key-Value (KV) cache poses a significant bottleneck, leading to memory overload and degraded inference throughput. A common compression method is to drop redundant KV tokens, which often breaks long-range dependencies, resulting in temporal flickering and identity loss. In this paper, we propose Instance-Specific Parametric Absorption (ISPA), a novel framework that shifts the KV cache compression from discarding to distilling. The core idea is to transit a subset of layers from Full-Attention (F-Layers) to memory-efficient Local-Attention (L-Layers) by "absorbing" historical context into the model's weights. Specifically, during a brief warmup phase, ISPA monitors the output discrepancy between global and local attention. At the transition point, we solve a closed-form least-squares problem to compute an instance-specific weight modulation that compensates for the missing history. Experiments across architectures (1.3B to 14B) demonstrate that ISPA can remove up to 50\% of the KV cache with near-lossless visual quality. We hope this perspective encourages future work to explore parametric memory consolidation beyond external token-level cache management for streaming generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。