用真实图片检索增强图像生成,减少幻觉,提升细节真实性。
RealRAG: Retrieval-augmented Realistic Image Generation via Self-reflective Contrastive Learning
- 通过自反思对比学习训练检索器,从真实图像中补全模型缺失知识。
- 在斯坦福汽车数据集上使自回归模型FID提升16.18%,显著改善生成质量。
- 可适配各类主流文生图模型,适合需要高精度实物生成的场景。
近期文生图模型(如Stable Diffusion V3和Flux)虽取得显著进展,但受限于封闭数据集训练的固定参数,面对细粒度及未见过的真实物体(如特斯拉Cybertruck外观)时仍存在严重幻觉或失真。为此,我们提出首个基于真实物体的检索增强生成框架RealRAG,通过学习并检索真实图像来弥补生成模型的知识缺口。具体地,我们采用自反思对比学习训练一个反射式检索器,将生成器的知识注入负样本,确保检索到的图像能有效补偿模型缺失信息。该框架引入细粒度视觉知识,缓解失真问题,提升生成真实性。RealRAG具备模块化设计,可兼容所有先进文生图模型,并在斯坦福汽车基准测试中为自回归模型带来16.18%的FID分数提升。
原文摘要 · Abstract (English)
Recent text-to-image generative models, e.g., Stable Diffusion V3 and Flux, have achieved notable progress. However, these models are strongly restricted to their limited knowledge, a.k.a., their own fixed parameters, that are trained with closed datasets. This leads to significant hallucinations or distortions when facing fine-grained and unseen novel real-world objects, e.g., the appearance of the Tesla Cybertruck. To this end, we present the first real-object-based retrieval-augmented generation framework (RealRAG), which augments fine-grained and unseen novel object generation by learning and retrieving real-world images to overcome the knowledge gaps of generative models. Specifically, to integrate missing memory for unseen novel object generation, we train a reflective retriever by self-reflective contrastive learning, which injects the generator's knowledge into the sef-reflective negatives, ensuring that the retrieved augmented images compensate for the model's missing knowledge. Furthermore, the real-object-based framework integrates fine-grained visual knowledge for the generative models, tackling the distortion problem and improving the realism for fine-grained object generation. Our Real-RAG is superior in its modular application to all types of state-of-the-art text-to-image generative models and also delivers remarkable performance boosts with all of them, such as a gain of 16.18% FID score with the auto-regressive model on the Stanford Car benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。