用小模型提升图像描述多样性，不需训练即可媲美大模型。
Image Embedding Sampling Method for Diverse Captioning
- 用结构化分割让小模型关注图像不同区域，生成更丰富描述。
- 在MSCOCO等数据集上多样性得分达0.735~0.750，接近大模型表现。
- 适合移动端和资源受限场景，无需额外训练，部署简单。
当前最先进的视觉语言模型（VLM）在图像描述任务上表现优异，但计算开销大，难以在移动设备和辅助技术等资源受限场景中应用。而较小的VLM虽轻量，却常忽略细节，仅提供高层语义。本文提出一种无需训练的框架，以小模型BLIP为骨干，通过结构化分割生成层次化表示，显式关注图像不同区域，提升描述的多样性和信息量。该方法无需额外训练，在MSCOCO、Flickr30k和Nocaps测试集上分别获得Div-2分数0.735、0.750和0.748，同时保持与人工标注描述相当的图像-文本相关性与语义完整性。
原文摘要 · Abstract (English)
Image Captioning for state-of-the-art VLMs has significantly improved over time; however, this comes at the cost of increased computational complexity, making them less accessible for resource-constrained applications such as mobile devices and assistive technologies. Alternatively, comparably smaller VLMs prioritize high-level scene descriptions, overlooking finer details that contribute to a richer understanding of an image. In this paper, we introduce a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM, BLIP, as the backbone. Our approach leverages structured segmentation to produce hierarchical representations that capture both global and localized semantics. Without requiring additional model training, we demonstrate that our method allows smaller VLMs to achieve performance comparable to larger models in terms of image-caption alignment, semantic integrity, and diversity. We evaluate our framework on MSCOCO, Flickr30k, and Nocaps test datasets, achieving a Div-2 score of 0.735, 0.750, and 0.748 for each dataset, respectively, while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。