无需训练即可实现逼真图像合成,解决光影与高分辨率难题
Does FLUX Already Know How to Perform Physically Plausible Image Composition?
- 用预训练适配器引导潜在空间,保持物体真实感且不破坏背景
- 在复杂光照下合成效果优于现有方法,人眼评分和模型指标均领先
- 适合需要高质量图像拼接的视觉设计、影视特效等场景
图像合成旨在将用户指定的物体无缝插入新场景,但现有模型在复杂光照(如准确阴影、水面反射)和多样高分辨率输入方面仍存在挑战。现代文生图扩散模型(如SD3.5、FLUX)已包含关键物理与分辨率先验,却缺乏有效框架来释放这些能力,传统方法依赖潜在空间反演,常导致物体姿态不符合上下文,或使用脆弱的注意力剪枝。本文提出SHINE——一种无需训练的无缝高保真插入框架,通过流形引导锚定损失,利用预训练定制适配器(如IP-Adapter)指导潜在表示以忠实再现主体,同时保持背景完整;引入退化抑制引导与自适应背景融合,进一步消除低质量输出与可见接缝。为弥补基准缺失,构建ComplexCompo数据集,涵盖多样化分辨率及低光、强光、复杂阴影、反射表面等挑战条件。在ComplexCompo与DreamEditBench上的实验表明,该方法在标准指标(如DINOv2)和人类对齐评分(如DreamSim、ImageReward、VisionReward)上均达到当前最优表现。代码已开源。
原文摘要 · Abstract (English)
Image composition aims to seamlessly insert a user-specified object into a new scene, but existing models struggle with complex lighting (e.g., accurate shadows, water reflections) and diverse, high-resolution inputs. Modern text-to-image diffusion models (e.g., SD3.5, FLUX) already encode essential physical and resolution priors, yet lack a framework to unleash them without resorting to latent inversion, which often locks object poses into contextually inappropriate orientations, or brittle attention surgery. We propose SHINE, a training-free framework for Seamless, High-fidelity Insertion with Neutralized Errors. SHINE introduces manifold-steered anchor loss, leveraging pretrained customization adapters (e.g., IP-Adapter) to guide latents for faithful subject representation while preserving background integrity. Degradation-suppression guidance and adaptive background blending are proposed to further eliminate low-quality outputs and visible seams. To address the lack of rigorous benchmarks, we introduce ComplexCompo, featuring diverse resolutions and challenging conditions such as low lighting, strong illumination, intricate shadows, and reflective surfaces. Experiments on ComplexCompo and DreamEditBench show state-of-the-art performance on standard metrics (e.g., DINOv2) and human-aligned scores (e.g., DreamSim, ImageReward, VisionReward). Code is available at https://github.com/ZhumingLian/SHINE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。