用多阶段流程提升高分辨率图像的精准描述,减少幻觉。
Generating Accurate and Detailed Captions for High-Resolution Images
- 结合VLM、LLM和目标检测,分步优化图像描述
- 新发现物体经检测后局部重写描述,细节更丰富
- 在高分辨率图像上显著降低幻觉,适合视觉分析场景
视觉语言模型(VLM)通常在低分辨率输入(如224x224或336x336像素)上预训练,难以生成高分辨率图像的准确详细描述。将高分辨率图像降采样会导致视觉细节丢失和重要物体遗漏。为此,我们提出一种新型多阶段流程,整合视觉语言模型、大语言模型(LLM)和目标检测系统以提升描述质量。给定一张高分辨率图像,先由VLM生成初始描述,再由LLM识别关键物体;随后,LLM预测与这些关键物体共现的其他可能物体,并通过目标检测系统验证。未在初始描述中提及的新检测到物体,会进行区域特异性重写,确保其被纳入。该过程提升了描述的细节性,同时通过移除未检测到物体的引用来减少幻觉。我们在一个精心筛选的高分辨率图像数据集上评估,采用成对比较、大型多模态模型的定量评分及幻觉检测基准。实验表明,该流程能生成更详细、更可靠的图像描述,有效抑制幻觉。
原文摘要 · Abstract (English)
Vision-language models (VLMs) often struggle to generate accurate and detailed captions for high-resolution images since they are typically pre-trained on low-resolution inputs (e.g., 224x224 or 336x336 pixels). Downscaling high-resolution images to these dimensions may result in the loss of visual details and the omission of important objects. To address this limitation, we propose a novel pipeline that integrates vision-language models, large language models (LLMs), and object detection systems to enhance caption quality. Our proposed pipeline refines captions through a novel, multi-stage process. Given a high-resolution image, an initial caption is first generated using a VLM, and key objects in the image are then identified by an LLM. The LLM predicts additional objects likely to co-occur with the identified key objects, and these predictions are verified by object detection systems. Newly detected objects not mentioned in the initial caption undergo focused, region-specific captioning to ensure they are incorporated. This process enriches caption detail while reducing hallucinations by removing references to undetected objects. We evaluate the enhanced captions using pairwise comparison and quantitative scoring from large multimodal models, along with a benchmark for hallucination detection. Experiments on a curated dataset of high-resolution images demonstrate that our pipeline produces more detailed and reliable image captions while effectively minimizing hallucinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。