提出可扩展的视频生成记忆检索框架，提升长视频生成质量与效率。
OmniMem: Scalable and Adaptive Memory Retrieval for Long Video Generation
- 采用稀疏关键值检索机制，从历史缓存中高效获取长期依赖信息。
- 在长视频生成中动态度提升52.3%，且保持强一致性与低内存开销。
- 适合需要高质量长序列生成的研究者，尤其关注视频生成与注意力优化。
自回归视频生成通过逐块生成潜空间序列来扩展视频，但生成长视频需反复访问不断增长的历史键值缓存。现有方法通过截断或压缩缓存来降低开销，但损失了对查询相关历史细节的显式访问。本文提出OmniMem，一种显式的全范围记忆检索框架，实现对历史缓存的稀疏键值（KV）检索。为适配基于块的自回归视频生成，OmniMem解决两个问题：(i) 稀疏键值选择中的局部偏差；(ii) 内存访问的并集爆炸。自适应窗口排除机制在有足够长程历史时，将局部窗口块排除在候选之外，使稀疏预算更聚焦于高信息量的远距离检索；查询共享键值选择减少跨查询多样性，每头分散键值访问避免头级选择膨胀为大缓存。这使得每个注意力头可根据自身模式检索非连续的键值块。在长视频生成实验中，OmniMem相比强基线动态度提升52.3%，同时保持强一致性且内存使用相当。
原文摘要 · Abstract (English)
Autoregressive (AR) video generation extends videos by producing latent chunks sequentially, but scaling to long videos requires repeated access to a growing historical KV cache. Existing methods reduce this cost by truncating the KV cache or compressing it into implicit memory, but both lose explicit access to query-relevant historical details. We propose OmniMem, an explicit full-range memory retrieval framework that performs sparse KV retrieval over the historical cache. To make this practical for chunk-based AR video generation, OmniMem addresses two issues: (i) local bias in sparse KV selection and (ii) Union Explosion in memory access. Adaptive Window Exclusion removes local-window blocks from the selection candidates when sufficient long-range history is available, preserving the sparse budget for informative long-range retrieval. Query-Shared KV Selection reduces cross-query diversity, while Per-Head Scattered KV Access avoids expanding head-specific selections into a large selected KV buffer. This allows each attention head to retrieve non-contiguous KV blocks according to its own selection pattern. Experiments on long-video generation show that OmniMem improves Dynamic Degree by 52.3% and preserves strong consistency over strong baselines, while maintaining comparable memory usage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。