用少10%数据实现更好解释性图像质量评估效果。
Exploring Instruction Data Quality for Explainable Image Quality Assessment
- 基于多层特征聚类,筛选高质量指令数据。
- 仅用10%数据达到全量训练102%-103%性能。
- 适合追求高效训练的AI模型开发者。
近年来,大型多模态模型(LMMs)的快速发展推动了可解释图像质量评估(IQA)的研究,旨在理解图像的感知质量问题。现有研究通常构建大规模指令微调数据集以提升LMM的感知能力,遵循数据规模定律。然而,随着LMM基础能力持续增强,现有指令数据集可能包含冗余且挑战性不足的样本,导致大量计算开销。本文探究在可解释IQA的LMM指令微调中,数据量是否仍是主导因素。基于强预训练的LMM,我们发现随机选取适量子集即可超越全量微调,表明现有数据集存在显著冗余。为此,我们提出Q-Selector——一种基于聚类的数据选择框架,包含三阶段:基于LMM的分层特征提取、依据密度与可迁移性分配聚类采样配额、以及基于SVD的聚类采样策略。Q-Selector从视觉编码器、文本编码器和大语言模型组件中提取多层特征作为聚类依据,根据簇间可迁移性和簇内密度确定采样数量,并采用奇异值分解(SVD)策略选取高质量指令数据。实验结果表明,在可解释IQA和图像美学评估任务上,仅使用10%训练数据,Q-Selector分别达到全量微调102.1%和103.7%的性能。
原文摘要 · Abstract (English)
In recent years, with the rapid development of large multimodal models (LMMs), explainable image quality assessment (IQA) has attracted increasing attention, aiming to understand the perceptual quality problems of images. Existing studies typically construct large-scale instruction tuning datasets to enhance the quality perception capabilities of LMMs, following the data scaling law. However, as the fundamental capabilities of LMMs continue to improve, existing instruction tuning datasets may contain redundant and less challenging samples, resulting in substantial computational costs. In this paper, we investigate whether data quantity remains the dominant factor in LMM instruction tuning for explainable IQA. Based on a strong pre-trained LMM, we observe that randomly selecting a properly sized subset of training data can outperform full-data fine-tuning, indicating substantial redundancy in existing instruction tuning datasets. Motivated by this observation, we propose Q-Selector, a clustering-based data selection framework consisting of three stages: hierarchical LMM-based clustering feature extraction, cluster quota allocation through density and transferability, and SVD-based cluster sampling strategy. Specifically, Q-Selector extracts multi-layer features from the vision encoder, text encoder, and large language model components of LMMs as clustering features. It then determines the sampling quota for each cluster based on inter-cluster transferability and intra-cluster density. Finally, Q-Selector employs a Singular Value Decomposition (SVD)-based sampling strategy to select high-quality instruction data. Experimental results demonstrate that Q-Selector achieves 102.1% and 103.7% of the performance of full-data fine-tuning using only 10% of the training data on explainable IQA and image aesthetics assessment tasks, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。