arXiv:2411.10161cs.CV2024-11被引 12

用视觉语言模型指导,实现图像关键区域的无参考质量评估。

SEAGULL: No-reference Image Quality Assessment for Regions of Interest via Vision-Language Instruction Tuning

  • 结合SAM掩码与视觉语言模型,提取区域级特征进行质量判断。
  • 在100万张合成图和3000个真实区域上训练,性能显著优于现有方法。
  • 适合需要精准修复图像局部质量问题的场景,如医学影像分析。

现有图像质量评估(IQA)方法在整体图像质量分析上已取得显著进展,但对关键区域(ROIs)的质量分析仍关注不足。区域级质量分析可为图像优化提供细粒度指导,在注重局部质量的场景中至关重要。本文提出一种新网络SEAGULL,能够通过大型视觉-语言模型(VLM)的引导,感知并评估图像中的关键区域质量。SEAGULL融合了视觉-语言模型、由分割任意模型(SAM)生成的掩码以指定区域,以及精心设计的基于掩码的特征提取器(MFE),用于提取指定区域的全局与局部特征,从而实现精准的细粒度ROI IQA。此外,本文构建了两个基于ROI的IQA数据集:SEAGULL-100w与SEAGULL-3k。其中,SEAGULL-100w包含约100万张合成失真图像和3300万条区域标注,用于预训练以提升模型对区域质量的感知能力;SEAGULL-3k包含约3000个真实失真区域,用于微调以增强模型对现实失真模式的识别能力。在SEAGULL-100w上预训练并在SEAGULL-3k上微调后,SEAGULL在细粒度区域质量评估任务中表现出色。代码与数据集已在https://github.com/chencn2020/Seagull公开。

原文摘要 · Abstract (English)

Existing Image Quality Assessment (IQA) methods achieve remarkable success in analyzing quality for overall image, but few works explore quality analysis for Regions of Interest (ROIs). The quality analysis of ROIs can provide fine-grained guidance for image quality improvement and is crucial for scenarios focusing on region-level quality. This paper proposes a novel network, SEAGULL, which can SEe and Assess ROIs quality with GUidance from a Large vision-Language model. SEAGULL incorporates a vision-language model (VLM), masks generated by Segment Anything Model (SAM) to specify ROIs, and a meticulously designed Mask-based Feature Extractor (MFE) to extract global and local tokens for specified ROIs, enabling accurate fine-grained IQA for ROIs. Moreover, this paper constructs two ROI-based IQA datasets, SEAGULL-100w and SEAGULL-3k, for training and evaluating ROI-based IQA. SEAGULL-100w comprises about 100w synthetic distortion images with 33 million ROIs for pre-training to improve the model's ability of regional quality perception, and SEAGULL-3k contains about 3k authentic distortion ROIs to enhance the model's ability to perceive real world distortions. After pre-training on SEAGULL-100w and fine-tuning on SEAGULL-3k, SEAGULL shows remarkable performance on fine-grained ROI quality assessment. Code and datasets are publicly available at the https://github.com/chencn2020/Seagull.

图像质量视觉语言区域评估无参考

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。