提前检测视频生成失败并干预,显著降低重试成本。
Early Failure Detection and Intervention in Video Diffusion Models
- 通过实时预览模块将隐变量转为中间视频,快速评估质量
- 在CogVideoX-5B上实现2.64倍提速,延迟仅39.2ms
- 可插拔兼容现有方法,适合高分辨率长视频生成
文本到视频扩散模型虽快速发展,但生成仍常失败,如文本对齐差或感知质量低。由于扩散采样非确定性,推理时难以预知生成成败,导致需反复重试,计算成本高。为此,我们提出针对潜空间视频扩散模型的早期失败检测与诊断干预流程。检测部分设计了实时检查(RI)模块,将隐变量转换为中间视频预览,使已有文本-视频对齐评分器可在RGB空间进行评估,该过程仅耗时39.2ms。相较之下,CogVideoX-5B在NVIDIA A100 GPU上生成480p、49帧视频时,每去噪步骤需4.3秒。当预测失败时,触发分层且早退出的干预流程。在CogVideoX-5B和Wan2.1-1.3B上的实验表明,相比事后重生成,VBench评测中一致性提升,时间开销减少达2.64倍。该方法还适用于更高容量模型,对Wan2.1-14B在720p、81帧生成中依然有效。此外,该流程可插拔且与现有技术正交,可无缝集成提示优化与采样引导方法。我们还证明失败信号在去噪早期即出现,可通过标准视觉语言评估器在中间预览中检测。
原文摘要 · Abstract (English)
Text-to-video (T2V) diffusion models have rapidly advanced, yet generations still occasionally fail in practice, such as low text-video alignment or low perceptual quality. Since diffusion sampling is non-deterministic, it is difficult to know during inference whether a generation will succeed or fail, incurring high computational cost due to trial-and-error regeneration. To address this, we propose an early failure detection and diagnostic intervention pipeline for latent T2V diffusion models. For detection, we design a Real-time Inspection (RI) module that converts latents into intermediate video previews, enabling the use of established text-video alignment scorers for inspection in the RGB space. The RI module completes the conversion and inspection process in just 39.2ms. This is highly efficient considering that CogVideoX-5B requires 4.3s per denoising step when generating a 480p, 49-frame video on an NVIDIA A100 GPU. Subsequently, we trigger a hierarchical and early-exit intervention pipeline only when failure is predicted. Experiments on CogVideoX-5B and Wan2.1-1.3B demonstrate consistency gains on VBench with up to 2.64 times less time overhead compared to post-hoc regeneration. Our method also generalizes to a higher-capacity setting, remaining effective on Wan2.1-14B with 720p resolution and 81-frame generation. Furthermore, our pipeline is plug-and-play and orthogonal to existing techniques, showing seamless compatibility with prompt refinement and sampling guidance methods. We also provide evidence that failure signals emerge early in the denoising process and are detectable within intermediate video previews using standard vision-language evaluators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。