用测试时训练让Transformer生成一分钟连贯长视频
One-Minute Video Generation with Test-Time Training
- 在预训练Transformer中加入可自学习的TTT层，提升长视频表达能力
- 生成视频故事连贯性领先基线34 Elo点，基于汤姆杰瑞动画数据集验证
- 适合关注长视频生成与动态隐藏状态建模的研究者
当前Transformer在生成一分钟视频时因自注意力机制对长序列效率低下而受限。替代方案如Mamba层在复杂多场景故事中表达力不足，因其隐状态表达能力有限。本文实验采用测试时训练（TTT）层，其隐状态本身为神经网络，具备更强表达能力。将TTT层融入预训练Transformer，使其能从文本分镜生成一分钟视频。为验证可行性，我们基于《汤姆和杰瑞》卡通制作了一个新数据集。相比Mamba~2、Gated DeltaNet及滑动窗口注意力等基线方法，TTT生成的视频在故事连贯性上显著更优，在每方法100个视频的人类评估中领先34 Elo点。尽管表现良好，结果仍存在伪影，可能源于预训练50亿参数模型能力有限。当前实现效率仍有优化空间。受资源限制，仅测试了一分钟视频，但该方法可扩展至更长视频与更复杂叙事。样本视频、代码与标注已公开于：https://test-time-training.github.io/video-dit
原文摘要 · Abstract (English)
Transformers today still struggle to generate one-minute videos because self-attention layers are inefficient for long context. Alternatives such as Mamba layers struggle with complex multi-scene stories because their hidden states are less expressive. We experiment with Test-Time Training (TTT) layers, whose hidden states themselves can be neural networks, therefore more expressive. Adding TTT layers into a pre-trained Transformer enables it to generate one-minute videos from text storyboards. For proof of concept, we curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers generate much more coherent videos that tell complex stories, leading by 34 Elo points in a human evaluation of 100 videos per method. Although promising, results still contain artifacts, likely due to the limited capability of the pre-trained 5B model. The efficiency of our implementation can also be improved. We have only experimented with one-minute videos due to resource constraints, but the approach can be extended to longer videos and more complex stories. Sample videos, code and annotations are available at: https://test-time-training.github.io/video-dit
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。