arXiv:2512.05198cs.CVcs.GR2025-12被引 5

提出像素等效潜空间融合,让扩散模型修复更精准无痕。

Your Latent Mask is Wrong: Pixel-Equivalent Latent Compositing for Diffusion Models

  • 设计DecFormer模型,实现潜空间与像素空间一致的融合
  • 修复后边缘误差降低53%,支持软边和全局色彩一致性
  • 可无缝接入现有扩散模型,轻量高效适合实际应用

扩散模型中的潜空间修复仍普遍依赖在下采样掩码上进行线性插值。本文提出像素等效潜空间融合(PELC)原则:潜空间融合应等价于像素空间融合。该原则使全分辨率掩码控制和真实软边透明度融合成为可能,即使VAE压缩了8倍空间分辨率。现代VAE捕捉到超出局部块对齐结构的全局上下文,因此线性潜空间混合无法实现像素等效,导致接缝处大尺度伪影、全局退化和颜色偏移。我们引入DecFormer——一个770万参数的Transformer,用于预测通道级混合权重及非流形残差修正,以实现掩码一致的潜空间融合。DecFormer训练目标为融合后解码结果匹配像素空间透明融合;它与现有扩散流程兼容,无需微调主干网络,仅增加FLUX.1-Dev 0.07%参数和3.5%计算开销。在FLUX.1系列上,DecFormer恢复了全局色彩一致性、软掩码支持、清晰边界和高保真修复,边缘误差降低最高达53%。作为修复先验,基于DecFormer的轻量LoRA在FLUX.1-Dev上达到与完全微调的FLUX.1-Fill相当的保真度。尽管聚焦于修复任务,PELC也可推广至其他像素等效潜空间编辑,如复杂色彩校正任务。

原文摘要 · Abstract (English)

Latent inpainting in diffusion models still relies almost universally on linearly interpolating VAE latents under a downsampled mask. We propose a key principle for compositing image latents: Pixel-Equivalent Latent Compositing (PELC). An equivalent latent compositor should be the same as compositing in pixel space. This principle enables full-resolution mask control and true soft-edge alpha compositing, even though VAEs compress images 8x spatially. Modern VAEs capture global context beyond patch-aligned local structure, so linear latent blending cannot be pixel-equivalent: it produces large artifacts at mask seams and global degradation and color shifts. We introduce DecFormer, a 7.7M-parameter transformer that predicts per-channel blend weights and an off-manifold residual correction to realize mask-consistent latent fusion. DecFormer is trained so that decoding after fusion matches pixel-space alpha compositing, is plug-compatible with existing diffusion pipelines, requires no backbone finetuning and adds only 0.07% of FLUX.1-Dev's parameters and 3.5% FLOP overhead. On the FLUX.1 family, DecFormer restores global color consistency, soft-mask support, sharp boundaries, and high-fidelity masking, reducing error metrics around edges by up to 53% over standard mask interpolation. Used as an inpainting prior, a lightweight LoRA on FLUX.1-Dev with DecFormer achieves fidelity comparable to FLUX.1-Fill, a fully finetuned inpainting model. While we focus on inpainting, PELC is a general recipe for pixel-equivalent latent editing, as we demonstrate on a complex color-correction task.

扩散模型图像修复潜空间融合像素等效

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。