无需训练即可提升文本生成视频的身份一致性,三步优化提示与图像。
Identity-Preserving Text-to-Video Generation via Training-Free Prompt, Image, and Guidance Enhancement
- 用GPT-4o从参考图提取面部细节增强文本提示。
- 通过身份保真图像生成器修正参考图与提示的冲突,提升输入质量。
- 采样时联合优化身份保留与视频质量,效果优于现有方法。
身份保真文本到视频(IPT2V)生成需同时忠实于参考主体图像和文本提示。尽管微调大模型可获最佳性能,但数据稀缺与高昂调优成本限制其推广。为此,我们提出无需训练的提示、图像与引导增强框架(TPIGE),弥合视频描述与参考图间的语义差距,并设计采样引导机制,共同提升身份保真度与视频质量,实现低成本性能提升。具体地,首先提出人脸感知提示增强,利用GPT-4o从参考图提取面部细节并增强文本提示;其次提出提示感知参考图增强,借助身份保真图像生成器修复参考图与文本提示间的矛盾;上述双向优化显著提升生成前输入质量。最后提出身份感知时空引导增强,采用统一梯度联合优化身份保真与视频质量。在1000视频测试集上,自动与人工评估均验证该方法优于现有工作,并在ACM Multimedia 2025身份保真视频生成挑战赛中夺得第一名,展现卓越性能与强泛化性。代码已开源:https://github.com/Andyplus1/IPT2V.git。
原文摘要 · Abstract (English)
Identity-preserving text-to-video (IPT2V) generation creates videos faithful to both a reference subject image and a text prompt. While fine-tuning large pretrained video diffusion models on ID-matched data achieves state-of-the-art results on IPT2V, data scarcity and high tuning costs hinder broader improvement. We thus introduce a Training-Free Prompt, Image, and Guidance Enhancement (TPIGE) framework that bridges the semantic gap between the video description and the reference image and design sampling guidance that enhances identity preservation and video quality, achieving performance gains at minimal cost.Specifically, we first propose Face Aware Prompt Enhancement, using GPT-4o to enhance the text prompt with facial details derived from the reference image. We then propose Prompt Aware Reference Image Enhancement, leveraging an identity-preserving image generator to refine the reference image, rectifying conflicts with the text prompt. The above mutual refinement significantly improves input quality before video generation. Finally, we propose ID-Aware Spatiotemporal Guidance Enhancement, utilizing unified gradients to optimize identity preservation and video quality jointly during generation.Our method outperforms prior work and is validated by automatic and human evaluations on a 1000 video test set, winning first place in the ACM Multimedia 2025 Identity-Preserving Video Generation Challenge, demonstrating state-of-the-art performance and strong generality. The code is available at https://github.com/Andyplus1/IPT2V.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。