用知识图谱增强图像掩码生成,提升细节质量
Improved Masked Image Generation with Knowledge-Augmented Token Representations
- 引入三类语义知识图谱作为先验信息
- 在ImageNet上实现更高保真度的条件图像生成
- 适配现有方法,轻量融合无额外训练开销
掩码图像生成(MIG)通过并行预测视觉标记序列,展现出高效与高保真特性。现有方法仅依赖模型自身学习视觉标记间的语义依赖关系,但因单个标记语义模糊且序列过长,学习困难。为此,本文提出知识增强型掩码图像生成框架KA-MIG,引入从训练数据中提取的标记级语义依赖显式知识作为先验,以学习更丰富的表示。具体探索并识别三类有益的知识图谱:共现图、语义相似图和位置-标记不相容图。基于此,设计图感知编码器以学习标记与位置感知的表示,并引入轻量级融合机制,将增强表示整合至现有MIG方法中。借助先验知识,本方法显著提升了模型捕捉语义依赖的能力,从而改善生成质量。实验表明,该方法在ImageNet上的类别条件图像生成任务中优于现有MIG方法。
原文摘要 · Abstract (English)
Masked image generation (MIG) has demonstrated remarkable efficiency and high-fidelity images by enabling parallel token prediction. Existing methods typically rely solely on the model itself to learn semantic dependencies among visual token sequences. However, directly learning such semantic dependencies from data is challenging because the individual tokens lack clear semantic meanings, and these sequences are usually long. To address this limitation, we propose a novel Knowledge-Augmented Masked Image Generation framework, named KA-MIG, which introduces explicit knowledge of token-level semantic dependencies (\emph{i.e.}, extracted from the training data) as priors to learn richer representations for improving performance. In particular, we explore and identify three types of advantageous token knowledge graphs, including two positive and one negative graphs (\emph{i.e.}, the co-occurrence graph, the semantic similarity graph, and the position-token incompatibility graph). Based on three prior knowledge graphs, we design a graph-aware encoder to learn token and position-aware representations. After that, a lightweight fusion mechanism is introduced to integrate these enriched representations into the existing MIG methods. Resorting to such prior knowledge, our method effectively enhances the model's ability to capture semantic dependencies, leading to improved generation quality. Experimental results demonstrate that our method improves upon existing MIG for class-conditional image generation on ImageNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。