无需训练,用检索增强生成理解图像质量。
QualiRAG: Retrieval-Augmented Generation for Visual Quality Understanding
- 通过动态构建四类辅助知识,实现细粒度视觉质量感知。
- 在多个任务上超越通用大模型和微调模型,且无需额外训练。
- 适合需要可解释图像质量评估的研究与应用者。
视觉质量评估(VQA)正从单一评分预测转向可解释的质量理解,这要求具备精细的时空感知能力与辅助上下文信息。现有方法依赖于人工标注的指令数据进行监督微调或强化学习,存在标注成本高、易受数据集偏差影响的问题。为此,我们提出训练免的RAG框架QualiRAG,系统性利用大模型的隐含感知知识进行视觉质量理解。不同于传统静态语料检索,QualiRAG将问题分解为结构化请求,构建四类互补知识源:视觉元数据、主体定位、全局质量摘要和局部质量描述,并通过相关性感知检索实现证据驱动推理。大量实验表明,QualiRAG在视觉质量理解任务上显著优于开源通用大模型及专用于VQA微调的大模型,在视觉质量对比任务中也表现竞争力,展现出无需任务特定训练的鲁棒质量评估能力。代码将公开于 https://github.com/clh124/QualiRAG。
原文摘要 · Abstract (English)
Visual quality assessment (VQA) is increasingly shifting from scalar score prediction toward interpretable quality understanding -- a paradigm that demands \textit{fine-grained spatiotemporal perception} and \textit{auxiliary contextual information}. Current approaches rely on supervised fine-tuning or reinforcement learning on curated instruction datasets, which involve labor-intensive annotation and are prone to dataset-specific biases. To address these challenges, we propose \textbf{QualiRAG}, a \textit{training-free} \textbf{R}etrieval-\textbf{A}ugmented \textbf{G}eneration \textbf{(RAG)} framework that systematically leverages the latent perceptual knowledge of large multimodal models (LMMs) for visual quality perception. Unlike conventional RAG that retrieves from static corpora, QualiRAG dynamically generates auxiliary knowledge by decomposing questions into structured requests and constructing four complementary knowledge sources: \textit{visual metadata}, \textit{subject localization}, \textit{global quality summaries}, and \textit{local quality descriptions}, followed by relevance-aware retrieval for evidence-grounded reasoning. Extensive experiments show that QualiRAG achieves substantial improvements over open-source general-purpose LMMs and VQA-finetuned LMMs on visual quality understanding tasks, and delivers competitive performance on visual quality comparison tasks, demonstrating robust quality assessment capabilities without any task-specific training. The code will be publicly available at https://github.com/clh124/QualiRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。