通过显式对齐参考图像特征与视觉基础模型语义空间,提升视频生成的身份一致性。
RefAlign: Representation Alignment for Reference-to-Video Generation
- 引入参考对齐损失,将参考图像特征与视觉基础模型特征在语义空间对齐。
- 在OpenS2V-Eval上总分超越现有最优方法,显著减少复制粘贴伪影。
- 训练阶段有效,推理无额外开销,适合个性化视频生成应用。
参考图像到视频(R2V)生成是一种可控视频合成范式,通过文本提示和参考图像共同约束生成过程,适用于个性化广告和虚拟试穿等场景。现有R2V方法通常在参考图像的VAE隐表示基础上引入高层语义或跨模态特征,并联合输入扩散Transformer(DiT)。这些辅助表示提供语义引导并作为隐式对齐信号,可部分缓解VAE隐空间中的像素级信息泄露问题。然而,它们仍难以解决由异构编码器特征间模态不匹配引发的复制粘贴伪影和多主体混淆问题。本文提出RefAlign,一种显式对齐DiT参考分支特征与视觉基础模型(VFM)语义空间的表示对齐框架。其核心是参考对齐损失:拉近同一主体的参考特征与VFM特征距离,提升身份一致性;推远不同主体对应特征距离,增强语义区分性。该策略仅在训练阶段使用,推理时无额外开销,实现了文本可控性与参考保真度之间的更好平衡。在OpenS2V-Eval基准上的大量实验表明,RefAlign在总分(TotalScore)上优于当前最先进方法,验证了显式参考对齐在R2V任务中的有效性。
原文摘要 · Abstract (English)
Reference-to-video (R2V) generation is a controllable video synthesis paradigm that constrains the generation process using both text prompts and reference images, enabling applications such as personalized advertising and virtual try-on. In practice, existing R2V methods typically introduce additional high-level semantic or cross-modal features alongside the VAE latent representation of the reference image and jointly feed them into the diffusion Transformer (DiT). These auxiliary representations provide semantic guidance and act as implicit alignment signals, which can partially alleviate pixel-level information leakage in the VAE latent space. However, they may still struggle to address copy--paste artifacts and multi-subject confusion caused by modality mismatch across heterogeneous encoder features. In this paper, we propose RefAlign, a representation alignment framework that explicitly aligns DiT reference-branch features to the semantic space of a visual foundation model (VFM). The core of RefAlign is a reference alignment loss that pulls the reference features and VFM features of the same subject closer to improve identity consistency, while pushing apart the corresponding features of different subjects to enhance semantic discriminability. This simple yet effective strategy is applied only during training, incurring no inference-time overhead, and achieves a better balance between text controllability and reference fidelity. Extensive experiments on the OpenS2V-Eval benchmark demonstrate that RefAlign outperforms current state-of-the-art methods in TotalScore, validating the effectiveness of explicit reference alignment for R2V tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。