让视频生成自动纠错,精准修复与文字不符的部分。
Self-Correcting Text-to-Video Generation with Misalignment Detection and Localized Refinement

- 通过自动生成问题检测视频与文本的错位区域
- 保留正确部分,仅重生成错误区域,提升一致性
- 无需训练,适配多种模型,适合复杂提示生成
近期文本到视频(T2V)扩散模型在生成高质量视频方面取得显著进展,但面对包含多个物体、属性或空间关系的复杂提示时,常出现对齐失败。我们提出 VideoRepair,首个无需训练、模型无关且具备自纠正能力的视频精修框架,能自动检测细粒度的文-视频错位并进行针对性局部修正。核心洞察是:即使整体错位的视频也包含正确生成的区域,应予以保留而非重生成。基于此,VideoRepair 提出一种区域保持式精修策略,包含三阶段:(i) 错位检测,利用基于多模态大模型的评估与自动生成评估问题识别错位区域;(ii) 精修规划,保留正确实体,跨帧分割其区域,并为错位区域构建目标提示;(iii) 局部精修,选择性重生成问题区域,通过联合优化保留内容与新生成内容实现一致修复。在 EvalCrafter 与 T2V-CompBench 两个基准上,使用四种最新 T2V 主干模型,VideoRepair 在多样对齐指标上均显著优于现有基线。全面消融实验进一步验证了该框架的高效性、鲁棒性与可解释性。
原文摘要 · Abstract (English)
Recent text-to-video (T2V) diffusion models have made remarkable progress in generating high-quality videos. However, they often struggle to align with complex text prompts, particularly when multiple objects, attributes, or spatial relations are specified. We introduce VideoRepair, the first self-correcting, training-free, and model-agnostic video refinement framework that automatically detects fine-grained text-video misalignments and performs targeted, localized corrections. Our key insight is that even misaligned videos usually contain correctly generated regions that should be preserved rather than regenerated. Building on this observation, VideoRepair proposes a novel region-preserving refinement strategy with three stages: (i) misalignment detection, where MLLM-based evaluation with automatically generated evaluation questions identifies misaligned regions; (ii) refinement planning, which preserves correctly generated entities, segments their regions across frames, and constructs targeted prompts for misaligned areas; and (iii) localized refinement, which selectively regenerates problematic regions while preserving faithful content through joint optimization of preserved and newly generated areas. On two benchmarks, EvalCrafter and T2V-CompBench with four recent T2V backbones, VideoRepair achieves substantial improvements over recent baselines across diverse alignment metrics. Comprehensive ablations further demonstrate the efficiency, robustness, and interpretability of our framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。