构建细粒度识别新基准，提升大模型视觉理解能力。
Towards Fine-Grained Recognition with Large Visual Language Models: Benchmark and Optimization Strategies
- 设计 mosaic 和 open-world 数据增强模型识别能力
- 引入新数据使识别准确率提升 10%-20%，内容准确率提升 6%-12%
- 适合关注视觉语言模型细节理解的研究者
大型视觉语言模型（LVLMs）在视觉-语言交互与对话应用中取得显著进展，但现有基准多聚焦于推理任务，忽视了对实际应用场景至关重要的细粒度识别。为此，我们提出了细粒度识别开放世界（FROW）基准，基于 GPT-4o 对 LVLM 进行全面评估。在此基础上，从数据构建与训练过程两个角度提出新型优化策略：数据包含拼贴式（mosaic）数据（整合多个短答案）和真实世界问答生成的开放世界数据；实验表明，mosaic 数据使类别识别准确率提升 1%，open-world 数据使 FROW 基准准确率提升 10%-20%，内容准确率提升 6%-12%；将细粒度数据融入预训练阶段，可使类别识别准确率最高提升 10%。基准代码与数据将开源。
原文摘要 · Abstract (English)
Large Vision Language Models (LVLMs) have made remarkable progress, enabling sophisticated vision-language interaction and dialogue applications. However, existing benchmarks primarily focus on reasoning tasks, often neglecting fine-grained recognition, which is crucial for practical application scenarios. To address this gap, we introduce the Fine-grained Recognition Open World (FROW) benchmark, designed for detailed evaluation of LVLMs with GPT-4o. On the basis of that, we propose a novel optimization strategy from two perspectives: \textit{data construction} and \textit{training process}, to improve the performance of LVLMs. Our dataset includes mosaic data, which combines multiple short-answer responses, and open-world data, generated from real-world questions and answers using GPT-4o, creating a comprehensive framework for evaluating fine-grained recognition in LVLMs. Experiments show that mosaic data improves category recognition accuracy by 1\% and open-world data boosts FROW benchmark accuracy by 10\%-20\% and content accuracy by 6\%-12\%. Meanwhile, incorporating fine-grained data into the pre-training phase can improve the model's category recognition accuracy by up to 10\%. The benchmark will be available at https://github.com/pc-inno/FROW.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。