用极少算力实现个性化视频生成,无需大量训练数据。
Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute
- 分解任务:从图像对学习身份特征,用少量视频保持动作连贯性。
- 仅需288小时A100算力,不到之前方法的3%。
- 适合想低成本定制视频生成模型的研究者和开发者。
主体驱动视频生成(SDV-Gen)旨在通过微调预训练视频模型生成特定主体的视频,实现个性化内容创作。传统方法需约200 A100 GPU小时进行每主体微调;而零样本方法虽免去微调,却依赖数百万主体-视频对,导致高达10,000–200,000 A100 GPU小时的网络微调成本。本文提出一种数据与算力高效零样本框架,避免测试时微调及大规模主体-视频对。核心思想将SDV-Gen分解为:(i) 从主体图像对中学习身份注入,(ii) 利用少量任意视频维持动作感知。通过随机参考帧采样与图像令牌丢弃进行随机切换优化,防止首帧简单复制。梯度分析表明,两项任务迅速演化至近乎正交更新子空间,解释了优化稳定性。基于CogVideoX-5B,仅用20万主体图像对与4,000个任意视频,在288 A100 GPU小时内完成单模型适配,计算量约为先前零样本基线的1%(即VACE的0.4%,Phantom的2.8%),且无需主体-视频对,仍保持高主体保真度与动作质量。该方法可迁移至Wan 2.2-5B。
原文摘要 · Abstract (English)
Subject-driven video generation (SDV-Gen) aims to produce videos of a specific subject by adapting a pretrained video model, enabling personalized and application-driven content creation. To achieve this goal, per-subject tuning methods require approximately 200 A100 GPU hours to generate a customized video, whereas zero-shot methods avoid per-subject tuning but typically rely on millions of subject-video pairs for the supervision, incurring massive network fine-tuning costs (10K-200K A100 GPU hours). We propose a data- and compute-efficient zero-shot SDV-Gen framework that avoids test-time per-subject tuning and the use of large-scale subject-video pairs. Our key idea decomposes SDV-Gen into (i) identity injection learned from subject-image pairs and (ii) motion-awareness preservation maintained by a small set of arbitrary videos. We optimize the two tasks with stochastic switching, using random reference-frame sampling and image-token dropout to prevent trivial first-frame copying. Our gradient analysis shows that the two objectives rapidly evolve toward nearly orthogonal update subspaces, explaining the stable optimization. Using CogVideoX-5B, we adapt a single model with 200K subject-image pairs and 4,000 arbitrary videos in 288 A100 GPU hours. This yields about 1% of compute compared to prior zero-shot baselines (i.e., 0.4% of VACE and 2.8% of Phantom) while using no subject-video pairs, yet remaining competitive in subject fidelity and motion quality. We show that the same recipe transfers to Wan 2.2-5B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。