发现视觉质量越高,模型表现反而越差,提出动态调优方法提升多模态模型理解力。
Demystifying the Visual Quality Paradox in Multimodal Large Language Models
- 通过可控降质与风格变换系统研究图像质量对模型影响
- 发现图像偏离人眼感知清晰度时,模型性能反而提升
- 提出轻量级VQ-TTT模块,无需额外数据即可动态优化输入
近期多模态大语言模型(MLLMs)在视觉-语言基准任务中表现优异,但其对输入图像质量的响应机制尚不明确。我们首次系统性研究了主流MLLMs在多种视觉-语言基准上的表现,对每张图像施加受控降质和风格变化。令人惊讶的是,我们发现了视觉质量悖论:当图像偏离人类感知的保真度时,模型在任务层面、模型层面甚至个别实例上的表现反而提升。现成的图像修复流程无法解决这种模型特有的偏好。为弥合差距,我们提出视觉质量测试时调优(VQ-TTT)——一种轻量级适配模块,包含:(1) 在冻结的视觉编码器前插入可学习的低秩核以调节频域内容;(2) 仅通过LoRA微调浅层视觉编码器层。VQ-TTT在单次前向传播中动态调整每张输入图像,使其匹配特定任务的模型偏好。在所有评估的MLLMs和数据集上,VQ-TTT均显著提升平均准确率,且无需外部模型、缓存特征或额外训练数据。这些发现重新定义了‘更好’的视觉输入,强调在人工智能成为主要数据消费者的新时代,应采用自适应而非普遍‘干净’的图像。
原文摘要 · Abstract (English)
Recent Multimodal Large Language Models (MLLMs) excel on benchmark vision-language tasks, yet little is known about how input visual quality shapes their responses. Does higher perceptual quality of images already translate to better MLLM understanding? We conduct the first systematic study spanning leading MLLMs and a suite of vision-language benchmarks, applying controlled degradations and stylistic shifts to each image. Surprisingly, we uncover a visual-quality paradox: model, task, and even individual-instance performance can improve when images deviate from human-perceived fidelity. Off-the-shelf restoration pipelines fail to reconcile these idiosyncratic preferences. To close the gap, we introduce Visual-Quality Test-Time Tuning (VQ-TTT)-a lightweight adaptation module that: (1) inserts a learnable, low-rank kernel before the frozen vision encoder to modulate frequency content; and (2) fine-tunes only shallow vision-encoder layers via LoRA. VQ-TTT dynamically adjusts each input image in a single forward pass, aligning it with task-specific model preferences. Across the evaluated MLLMs and all datasets, VQ-TTT lifts significant average accuracy, with no external models, cached features, or extra training data. These findings redefine ``better'' visual inputs for MLLMs and highlight the need for adaptive, rather than universally ``clean'', imagery, in the new era of AI being the main data customer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。