让大模型同时学会评价图像质量并解释原因,提升智能评估能力。
Teaching LMMs for Image Quality Scoring and Interpreting
- 将图像质量评估任务转为问答形式,联合训练模型评分与解释能力。
- 在多个数据集上实现领先性能,评分与解释均表现优异。
- 提出高效平衡策略,降低计算成本,适合实际部署应用。
图像质量评分与解释是图像质量评估(IQA)的两大核心任务:前者量化图像质量,后者支持关于质量的描述性问答。传统方法独立处理二者,但基于人眼视觉系统(HVS)和感知-决策整合模型,两者本质关联:解释是评分的基础,评分则是解释的抽象总结。为此,本文提出Q-SiT(Quality Scoring and Interpreting joint Teaching),一个统一框架,使大视觉语言模型(LMMs)能同时学习评分与解释。通过将传统IQA数据集转化为可学习的问答数据,并融合人工标注的质量解释数据进行训练,实现联合学习。进一步提出一种高效评分与解释平衡策略:先在轻量级LMM上确定最优数据混合比例,再映射至主模型进行微调。该策略有效缓解任务干扰,增强跨任务知识迁移,显著降低计算开销。实验表明,Q-SiT在多项指标上表现卓越,具备强泛化能力;其轻量版本Q-SiT-mini亦表现良好。项目页面见https://github.com/Q-Future/Q-SiT。
原文摘要 · Abstract (English)
Image quality scoring and interpreting are two fundamental components of Image Quality Assessment (IQA). The former quantifies image quality, while the latter enables descriptive question answering about image quality. Traditionally, these two tasks have been addressed independently. However, from the perspective of the Human Visual System (HVS) and the Perception-Decision Integration Model, they are inherently interconnected: interpreting serves as the foundation for scoring, while scoring provides an abstract summary of interpreting. Thus, unifying these capabilities within a single model is both intuitive and logically coherent. In this paper, we propose Q-SiT (Quality Scoring and Interpreting joint Teaching), a unified framework that enables large multimodal models (LMMs) to learn both image quality scoring and interpreting simultaneously. We achieve this by transforming conventional IQA datasets into learnable question-answering datasets and incorporating human-annotated quality interpreting data for training. Furthermore, we introduce an efficient scoring & interpreting balance strategy, which first determines the optimal data mix ratio on lightweight LMMs and then maps this ratio to primary LMMs for fine-tuning adjustment. This strategy not only mitigates task interference and enhances cross-task knowledge transfer but also significantly reduces computational costs compared to direct optimization on full-scale LMMs. With this joint learning framework and corresponding training strategy, we develop Q-SiT, the first model capable of simultaneously performing image quality scoring and interpreting tasks, along with its lightweight variant, Q-SiT-mini. Experimental results demonstrate that Q-SiT achieves strong performance in both tasks with superior generalization IQA abilities.Project page at https://github.com/Q-Future/Q-SiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。