arXiv:2507.14367cs.CV2025-07被引 7

用大模型评分生成图像中的幻觉问题,提升真实感。

Hallucination Score: Towards Mitigating Hallucinations in Generative Image Super-Resolution

  • 用多模态大模型设计提示词,量化生成细节与低分辨率图的不一致程度。
  • 新评分与人类评估高度一致,且补充了传统图像指标的不足。
  • 可作为可微分奖励,指导扩散模型减少幻觉,适合图像修复与生成研究者。

生成式超分辨率(GSR)在感知质量上达到当前最优,克服了以往非生成模型的“回归均值”模糊问题。然而从人类视角看,这类模型在质量与保真度之间未达最佳平衡。一种关键但被忽视的缺陷是:生成的细节无法在感知上匹配低分辨率图像(LRI)或真实图像(GTI),此类现象称为“幻觉”。现有图像指标和质量模型难以有效刻画这类幻觉,因其与精确保真度和无参考质量均无关。本文提出利用多模态大语言模型(MLLM),构建评估幻觉视觉元素的提示,生成“幻觉评分”(HS)。实验表明,HS与人类评价高度一致,并为超分辨率评估提供互补信息。我们进一步设计了高效的HS代理指标,将其作为可微分奖励函数,用于微调基于扩散的GSR模型,有效缓解幻觉问题。

原文摘要 · Abstract (English)

Generative super-resolution (GSR) currently sets the state-of-the-art in terms of perceptual image quality, overcoming the "regression-to-the-mean" blur of prior non-generative models. However, from a human perspective, such models do not fully conform to the optimal balance between quality and fidelity. Instead, a different class of artifacts, in which generated details fail to perceptually match the low resolution image (LRI) or ground-truth image (GTI), is a critical but under-studied issue in GSR, limiting its practical deployment. In this work, we focus on measuring, analyzing, and mitigating these artifacts (i.e., "hallucinations"). We observe that hallucinations are not well-characterized with existing image metrics or quality models, as they are orthogonal to both exact fidelity and no-reference quality. Instead, we take advantage of multimodal large language models (MLLMs) by constructing a prompt that assesses hallucinatory visual elements and generates a "Hallucination Score" (HS). We find that HS is closely aligned with human evaluations, and also provides complementary insights to prior image metrics used for super-resolution (SR) models. Finally, we propose a few efficient HS proxies and demonstrate how diffusion-based GSR models can be fine-tuned to mitigate hallucinations, leveraging HS proxies as differentiable reward functions.

图像超分辨率幻觉检测多模态大模型扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。