让文字生成视频更精准，通过识别视频中每个物体来提升描述质量。
InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption
- 基于实例感知的结构化描述框架，精细刻画视频中每个物体。
- 在22,000个视频上训练，显著降低幻觉并提高生成视频与描述的一致性。
- 适合需要高精度图文对齐的应用，如影视创作和智能视频编辑。
近年来，文本到视频生成技术发展迅速，取得了显著成果。训练通常依赖于视频-字幕配对数据，这对提升生成性能至关重要。然而，现有视频字幕常缺乏细节、存在幻觉且运动描述不准确，影响生成视频的保真度与一致性。本文提出一种新型实例感知的结构化字幕框架——InstanceCap，首次实现视频中实例级、细粒度的字幕生成。基于该框架，设计辅助模型集群将原始视频分解为实例，以增强实例保真度；视频实例进一步用于将密集提示精炼为结构化短语，实现简洁而精确的描述。此外，构建了包含22,000个样本的InstanceVid数据集，并提出了针对InstanceCap结构优化的推理增强流水线。实验表明，所提方法显著优于现有模型，在保持字幕与视频高度一致的同时，有效减少幻觉现象。
原文摘要 · Abstract (English)
Text-to-video generation has evolved rapidly in recent years, delivering remarkable results. Training typically relies on video-caption paired data, which plays a crucial role in enhancing generation performance. However, current video captions often suffer from insufficient details, hallucinations and imprecise motion depiction, affecting the fidelity and consistency of generated videos. In this work, we propose a novel instance-aware structured caption framework, termed InstanceCap, to achieve instance-level and fine-grained video caption for the first time. Based on this scheme, we design an auxiliary models cluster to convert original video into instances to enhance instance fidelity. Video instances are further used to refine dense prompts into structured phrases, achieving concise yet precise descriptions. Furthermore, a 22K InstanceVid dataset is curated for training, and an enhancement pipeline that tailored to InstanceCap structure is proposed for inference. Experimental results demonstrate that our proposed InstanceCap significantly outperform previous models, ensuring high fidelity between captions and videos while reducing hallucinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。