arXiv:2504.20438cs.CV2025-04被引 7

用结构语义引导修复图像,减少错乱和不一致。

PixelHacker: Image Inpainting with Structural and Semantic Consistency

  • 用嵌入向量分别表示前景背景,通过线性注意力注入去噪过程。
  • 在1400万对图像上预训练,在多个数据集上超越现有最佳方法。
  • 适合需要高结构语义一致性的图像修复场景。

图像修复是图像编辑与生成之间的基础研究方向。近期最先进方法探索了新型注意力机制、轻量级架构和上下文感知建模,表现优异。但通常在复杂结构(如纹理、形状、空间关系)和语义(如颜色一致性、物体恢复、逻辑正确性)上仍存在问题,导致伪影和不恰当生成。为此,我们设计了一种简单而有效的修复范式:潜在类别引导,并提出基于扩散模型的PixelHacker。具体而言,我们首先通过标注前景(116类)和背景(21类)构建包含1400万张图像-掩码对的大规模数据集;接着使用两个固定大小的嵌入分别编码前景与背景表征,并通过线性注意力间歇性注入去噪过程;最后在该数据集上预训练并微调至开源基准,得到PixelHacker。大量实验表明,PixelHacker在多个数据集(Places2、CelebA-HQ、FFHQ)上全面超越现有最先进方法,且在结构与语义上均表现出显著一致性。

原文摘要 · Abstract (English)

Image inpainting is a fundamental research area between image editing and image generation. Recent state-of-the-art (SOTA) methods have explored novel attention mechanisms, lightweight architectures, and context-aware modeling, demonstrating impressive performance. However, they often struggle with complex structure (e.g., texture, shape, spatial relations) and semantics (e.g., color consistency, object restoration, and logical correctness), leading to artifacts and inappropriate generation. To address this challenge, we design a simple yet effective inpainting paradigm called latent categories guidance, and further propose a diffusion-based model named PixelHacker. Specifically, we first construct a large dataset containing 14 million image-mask pairs by annotating foreground and background (potential 116 and 21 categories, respectively). Then, we encode potential foreground and background representations separately through two fixed-size embeddings, and intermittently inject these features into the denoising process via linear attention. Finally, by pre-training on our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker. Extensive experiments show that PixelHacker comprehensively outperforms the SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits remarkable consistency in both structure and semantics. Project page at https://hustvl.github.io/PixelHacker.

图像修复扩散模型结构一致性语义修复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。