让AI画画时同时听懂文字和参考图，不需重新训练。
Injecting Image Guidance into Text-Conditioned Diffusion Models at Inference
- 通过特征对齐，让图像与文本在同一个空间中融合。
- 能准确传递参考图的风格、构图和配色，且不偏离文字描述。
- 适合需要快速调图风格的设计师或内容创作者使用。
如Stable Diffusion等文生图扩散模型虽能生成高质量图像，但无法在推理阶段注入视觉引导（如草图、风格）而不进行重训练。现有方法要么计算开销大，要么依赖风格迁移导致语义偏差。本文提出视觉概念融合（VCF），首次实现推理时对图像和文本双条件控制，无需特定概念训练。VCF通过将CLIP图像特征对齐至文本嵌入空间，包含三个部分：(1) 轻量级对齐器，利用InfoNCE与交叉注意力重建损失将图像标记映射到文本嵌入流形；(2) 融合策略，保留文本与视觉语义；(3) 可选的测试时优化模块（PNO）。实验表明，VCF能有效转移参考图的风格、构图与色彩，同时保持对提示的遵循。定量结果显示，在文本一致性（CLIP分数）与视觉对应性（LPIPS）间存在权衡，但VCF在参考图保真度上优于基线。
原文摘要 · Abstract (English)
Text-to-image diffusion models like Stable Diffusion generate high-quality images from text, but lack a way to inject visual guidance (e.g. sketches, styles) at inference without retraining. Existing methods either require computationally expensive fine-tuning or rely on style transfer techniques that risk semantic misalignment with textual prompts. We introduce Visual Concept Fusion (VCF), the first method offering dual conditioning on both an image and text prompt at inference time without any concept-specific training. VCF enables visual concept injection into Stable Diffusion by aligning CLIP image features with the text embedding space. VCF consists of three components: (1) a lightweight aligner that maps image tokens to the text embedding manifold using InfoNCE and cross-attention reconstruction losses, (2) a fusion strategy that preserves both textual and visual semantics, and (3) an optional Prompt-Noise Optimization (PNO) module for test-time refinement. Our experiments demonstrate that VCF successfully transfers visual attributes including style, composition, and color palette from reference images while maintaining prompt adherence. Quantitative results show a trade-off between text alignment (CLIP score) and visual correspondence (LPIPS), with VCF outperforming baselines in reference fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。