arXiv:2511.11410cs.CV2025-11被引 1

评测大模型对文档图像质量的感知能力,发现其评分不准且易误判缺陷。

Q-Doc: Benchmarking Document Image Quality Assessment Capabilities in Multi-modal Large Language Models

  • 构建三级评估框架,从粗到细测试模型对文档质量的判断能力
  • 模型在严重度判断上准确率不足60%,且对多种失真类型识别错误率高
  • 使用思维链提示可显著提升各层级表现,适合想改进文档理解的开发者

多模态大模型(MLLMs)虽在高级视觉任务中表现优异,但其在文档图像质量评估(DIQA)方面的潜力仍待挖掘。为此,我们提出Q-Doc,一个三层次评估框架,系统评测MLLMs在粗粒度、中粒度和细粒度上的DIQA能力:在粗粒度层面,要求模型为文档图像打分并分析与人工标注的相关性;在中粒度层面,设计单选与多选失真类型识别任务,覆盖多种失真场景;在细粒度层面,引入失真严重度评估,要求模型根据人类标注参考进行强度分类。评估表明,尽管模型具备初步的感知能力,但仍存在评分不一致、失真类型误判及严重度误判等关键缺陷。值得注意的是,采用思维链(CoT)提示后,模型在所有层级的表现均有显著提升。本研究提供了一个可用于评估MLLMs DIQA能力的基准,揭示了其在质量感知方面的明显不足,并指明了优化路径。基准代码已公开于:https://github.com/cydxf/Q-Doc。

原文摘要 · Abstract (English)

The rapid advancement of Multi-modal Large Language Models (MLLMs) has expanded their capabilities beyond high-level vision tasks. Nevertheless, their potential for Document Image Quality Assessment (DIQA) remains underexplored. To bridge this gap, we propose Q-Doc, a three-tiered evaluation framework for systematically probing DIQA capabilities of MLLMs at coarse, middle, and fine granularity levels. a) At the coarse level, we instruct MLLMs to assign quality scores to document images and analyze their correlation with Quality Annotations. b) At the middle level, we design distortion-type identification tasks, including single-choice and multi-choice tests for multi-distortion scenarios. c) At the fine level, we introduce distortion-severity assessment where MLLMs classify distortion intensity against human-annotated references. Our evaluation demonstrates that while MLLMs possess nascent DIQA abilities, they exhibit critical limitations: inconsistent scoring, distortion misidentification, and severity misjudgment. Significantly, we show that Chain-of-Thought (CoT) prompting substantially enhances performance across all levels. Our work provides a benchmark for DIQA capabilities in MLLMs, revealing pronounced deficiencies in their quality perception and promising pathways for enhancement. The benchmark and code are publicly available at: https://github.com/cydxf/Q-Doc.

文档质量多模态大模型评测思维链

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。