用分布建模让大模型更准地打图像质量分。
Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution
- 用软标签替代硬标签,保留分数分布特性
- 跨数据集联合训练,提升评分稳定性
- 适合需要精准图像质量评估的场景
随着多模态大模型(MLLM)的快速发展,基于MLLM的图像质量评估方法在语言描述质量方面展现出良好性能。然而,现有方法在准确评分上仍存在不足。本文旨在利用MLLM回归精确的质量分数。核心挑战在于质量分数是连续的,通常服从高斯分布,而MLLM生成的是离散标记。此前方法将均值离散化为独热编码,造成信息损失且无法捕捉图像间关系。我们提出一种基于分布的软标签方法,对分数分布进行离散化,有效保留分布特征并维持图像间关系。此外,针对不同IQA数据集分布差异的问题,引入基于Thurstone模型的保真度损失,捕捉数据集内关系,支持多数据集协同训练。基于此,我们构建了分布式图像质量评分模型DeQA-Score。在多个基准测试中,DeQA-Score稳定优于基线模型,且预测的分数分布与人工标注高度一致。代码与模型权重已公开于https://depictqa.github.io/deqa-score/。
原文摘要 · Abstract (English)
With the rapid advancement of Multi-modal Large Language Models (MLLMs), MLLM-based Image Quality Assessment (IQA) methods have shown promising performance in linguistic quality description. However, current methods still fall short in accurately scoring image quality. In this work, we aim to leverage MLLMs to regress accurate quality scores. A key challenge is that the quality score is inherently continuous, typically modeled as a Gaussian distribution, whereas MLLMs generate discrete token outputs. This mismatch necessitates score discretization. Previous approaches discretize the mean score into a one-hot label, resulting in information loss and failing to capture inter-image relationships. We propose a distribution-based approach that discretizes the score distribution into a soft label. This method preserves the characteristics of the score distribution, achieving high accuracy and maintaining inter-image relationships. Moreover, to address dataset variation, where different IQA datasets exhibit various distributions, we introduce a fidelity loss based on Thurstone's model. This loss captures intra-dataset relationships, facilitating co-training across multiple IQA datasets. With these designs, we develop the distribution-based Depicted image Quality Assessment model for Score regression (DeQA-Score). Experiments across multiple benchmarks show that DeQA-Score stably outperforms baselines in score regression. Also, DeQA-Score can predict the score distribution that closely aligns with human annotations. Codes and model weights have been released in https://depictqa.github.io/deqa-score/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。