通过锚定字幕生成优化提示,提升文本生成视频的一致性。
CAPE-T2V: Captioner-Anchored Prompt Enhancement toward Two-Sided Conditioning Alignment in Text-to-Video Generation

- 用字幕生成目标反向训练提示增强器,对齐训练与推理输入。
- 在多个评测集上优于基线模型,尤其在叙事连贯性上提升显著。
- 适合关注生成一致性与提示工程的视频生成研究者。
文本到视频扩散变换器(DiTs)在训练时使用详细视频字幕,而推理时依赖由提示增强器(PE)重写的用户提示。现有方法通过优化PE、DiT或两者来改进生成效果,部分工作尝试通过共享结构缩小训练-推理差异。然而即使在共享结构下,推理时的PE输出与训练字幕仍可能存在细节选择、信息组织、描述粒度和表述方式的差异。本文将此残余差异称为PE-字幕差距,提出CAPE-T2V框架,通过两阶段实现双向条件对齐:第一阶段构建三类训练样本,将字幕生成的目标与简短源字幕、详细源字幕或由目标推导出的伪用户提示配对,微调PE使其映射至对应目标;第二阶段用经锚定后的PE重写的视频字幕微调DiT,推理时同样使用该PE重写用户提示。相比采用相同字幕结构的基线,CAPE-T2V在Wan2.2和LTX-2.3上的StoryEval、VBench-2.0和T2V-CompBench均获得更高综合得分。此外,其微调后的字幕分布与推理时的PE输出更接近,最大均值差异平方(MMD²)更低。结果表明,该方法能有效缓解PE-字幕差距。项目地址:https://github.com/yizzz927/CAPE-T2V。
原文摘要 · Abstract (English)
Text-to-video (T2V) diffusion transformers (DiTs) are trained with detailed video captions, whereas inference often relies on user prompts rewritten by a prompt enhancer (PE). Prior work has improved generation by optimizing the PE, the DiT, or both; some methods have also sought to narrow the training-inference mismatch through shared schemas. Yet even within a shared schema, inference-time PE outputs and DiT training captions may still differ in detail selection, information organization, descriptive granularity, and phrasing. We refer to this residual mismatch as the PE-Caption gap and introduce CAPE-T2V, a two-step Captioner-Anchored Prompt Enhancement framework toward two-sided conditioning alignment in T2V generation. First, CAPE-T2V constructs three types of PE training examples, pairing captioner-generated targets with concise source captions, detailed source captions, or pseudo user prompts derived from those targets. It then fine-tunes the PE to map each input to its paired target. Second, CAPE-T2V fine-tunes the DiT on video-derived captions rewritten by the Anchored PE; the same PE rewrites user prompts at inference. Relative to a baseline using the same caption schema, CAPE-T2V achieves higher aggregate scores on StoryEval, VBench-2.0, and T2V-CompBench across Wan2.2 and LTX-2.3. Further, CAPE-T2V exhibits a smaller PE-Caption gap than the baseline: its DiT fine-tuning captions are closer in distribution to inference-time PE outputs, as measured by squared maximum mean discrepancy in a fixed embedding space. Overall, these results support CAPE-T2V as an effective approach to mitigating the PE-Caption gap. The project is available at https://github.com/yizzz927/CAPE-T2V.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。