用超像素+视觉语言模型，让图像描述摆脱对目标检测的依赖。
SuperCap: Multi-resolution Superpixel-based Image Captioning
- 用超像素提取类物体特征，替代传统目标检测。
- 在COCO数据集上达到136.9的CIDEr分数，表现优秀。
- 支持多分辨率输入，适合需要细节与整体结合的应用。
长期以来，图像描述任务依赖目标检测。本文提出基于超像素的新型方法，结合视觉语言模型（VLMs），在不使用目标检测的前提下实现类物体特征输入，并借助VLMs获得开放集物体理解能力。进一步，模型采用多分辨率输入机制，通过注意力机制动态选择关键区域生成描述。我们在多种VLM架构上验证了方法有效性，并通过消融实验分析不同设计的影响。最终模型在COCO Karpathy测试集上取得136.9的CIDEr得分，表现优异。
原文摘要 · Abstract (English)
It has been a longstanding goal within image captioning to move beyond a dependence on object detection. We investigate using superpixels coupled with Vision Language Models (VLMs) to bridge the gap between detector-based captioning architectures and those that solely pretrain on large datasets. Our novel superpixel approach ensures that the model receives object-like features whilst the use of VLMs provides our model with open set object understanding. Furthermore, we extend our architecture to make use of multi-resolution inputs, allowing our model to view images in different levels of detail, and use an attention mechanism to determine which parts are most relevant to the caption. We demonstrate our model's performance with multiple VLMs and through a range of ablations detailing the impact of different architectural choices. Our full model achieves a competitive CIDEr score of $136.9$ on the COCO Karpathy split.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。