轻量级视频历史编码，提升生成一致性且节省资源
TinyHistory: Lightweight Video History Embeddings via Two-Stage Context Learning
- 分两阶段学习：先随机帧查询预训练，再在扩散模型中优化内容一致性
- 在有限内存下可编码更长历史，性能接近重型方案
- 适合个人使用、离线流程及小规模微调场景
历史上下文对自回归视频生成至关重要，有助于保持一致性和叙事连贯性。个人用户、离线工作流和小规模微调场景需在计算与内存受限条件下编码更长的视频历史。我们观察到内容与身份一致性是核心需求，且希望完整覆盖历史并具备内容查询与理解能力。本文提出TinyHistory，一种通过两阶段上下文学习获得的轻量级历史嵌入。第一阶段，在大规模视频数据上以随机帧查询目标预训练编码器；第二阶段，将预训练编码器引入自回归视频扩散模型中，学习内容层面的一致性。结果表明，该轻量级嵌入在VLM、VBench、ELO等指标上表现媲美重型方案，同时降低训练开销，并在固定内存预算内扩展可编码历史长度。我们进行了消融实验，分析各组件的影响与权衡。
原文摘要 · Abstract (English)
History context is central to autoregressive video generation, driving consistency and storytelling for both commercial models and personal use cases. For example, personal users, offline workflows, and individual-scale finetuning need to encode longer video histories under tight compute and memory budgets. We observe that content and identity consistency is an essential requirement, and that complete, uninterrupted history coverage together with content query and interpretation capabilities is broadly desired. We present TinyHistory, a lightweight history embedding learned through two-stage context learning. In the first stage, we pretrain the encoder on large-scale video data with a randomized frame query objective; in the second stage, we repurpose the pretrained encoder within an autoregressive video diffusion model to learn content-level consistency. As a result, we show that the learned lightweight embeddings achieve consistency comparable (by VLM, VBench, ELO, etc) to heavier alternatives, while reducing training overhead and extending the encodable history length within a given memory budget. We conduct ablation studies to analyze the influence and trade-offs of each component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。