统一视觉条件融合,让多参考图像生成更准确保真。
UniCustom: Unified Visual Conditioning for Multi-Reference Image Generation

- 早融合ViT与VAE特征,让模型同时理解语义和外观细节。
- 在两个基准上显著提升主体一致性与指令遵循能力。
- 适合需要精准保留参考图像特征的研究者使用。
多参考图像生成旨在根据文本指令合成图像,同时忠实保留多个参考图像中的主体身份。现有基于视觉语言模型(VLM)的扩散模型通常采用解耦视觉条件机制:语义ViT特征由VLM处理以理解指令,而富含外观信息的VAE特征则后期注入扩散主干。这种分离导致模型难以将每个语义主体与其对应的参考图像外观关联,造成属性泄露和跨参考混淆。为此,我们提出UniCustom,一种在VLM编码前融合ViT与VAE特征的统一视觉条件框架。该早期融合使VLM的隐藏状态能联合编码被指代主体及其对应外观,仅需轻量线性融合层。为学习此类统一表征,采用两阶段训练策略:以重建为导向的预训练保留融合隐藏状态中的参考特异性外观细节,随后在单/多参考生成任务上进行监督微调。进一步引入槽位绑定正则化,促使每个图像槽保留其对应参考的低级细节,减少跨参考纠缠。在两个多参考生成基准上的实验表明,UniCustom在主体一致性、指令遵循和组合保真度方面均持续优于强基线。
原文摘要 · Abstract (English)
Multi-reference image generation aims to synthesize images from textual instructions while faithfully preserving subject identities from multiple reference images. Existing VLM-enhanced diffusion models commonly rely on decoupled visual conditioning: semantic ViT features are processed by the VLM for instruction understanding, whereas appearance-rich VAE features are injected later into the diffusion backbone. Despite its intuitive design, this separation makes it difficult for the model to associate each semantically grounded subject with visual details from the correct reference image. As a result, the model may recognize which subject is being referred to, but fail to preserve its identity and fine-grained appearance, leading to attribute leakage and cross-reference confusion in complex multi-reference settings. To address this issue, we propose UniCustom, a unified visual conditioning framework that fuses ViT and VAE features before VLM encoding. This early fusion exposes the VLM to both semantic cues and appearance-rich details, enabling its hidden states to jointly encode the referred subject and corresponding visual appearance with only a lightweight linear fusion layer. To learn such unified representations, we adopt a two-stage training strategy: reconstruction-oriented pretraining that preserves reference-specific appearance details in the fused hidden states, followed by supervised finetuning on single- and multi-reference generation tasks. We further introduce a slot-wise binding regularization that encourages each image slot to preserve low-level details of its corresponding reference, thereby reducing cross-reference entanglement. Experiments on two multi-reference generation benchmarks demonstrate that UniCustom consistently improves subject consistency, instruction following, and compositional fidelity over strong baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。