ViTOC用双路结构融合全局与局部视觉信息，提升图像描述准确性和多样性。
ViTOC: Vision Transformer and Object-aware Captioner
- 双路径架构结合Vision Transformer与目标检测器，用可学习向量融合全局与局部特征。
- 在COCO数据集上所有评估指标均优于基线模型，尤其对长尾物体描述效果显著。
- 适合需要高质量、多样化图像描述的场景，如内容生成与视觉理解系统。
本文提出ViTOC（Vision Transformer and Object-aware Captioner），一种新型视觉语言模型，用于图像描述生成，旨在解决生成描述的准确性与多样性挑战。不同于传统方法，ViTOC采用基于Vision Transformer和目标检测器的双路径架构，通过可学习向量有效融合全局视觉特征与局部物体信息。模型引入创新的物体感知提示策略，显著提升对长尾数据的处理能力。在标准COCO数据集上的实验表明，ViTOC在所有评估指标上均优于基线模型。此外，我们提出一种基于CLIP的无参考评估方法，进一步验证模型有效性。通过利用预训练视觉模型参数，ViTOC实现高效端到端训练。
原文摘要 · Abstract (English)
This paper presents ViTOC (Vision Transformer and Object-aware Captioner), a novel vision-language model for image captioning that addresses the challenges of accuracy and diversity in generated descriptions. Unlike conventional approaches, ViTOC employs a dual-path architecture based on Vision Transformer and object detector, effectively fusing global visual features and local object information through learnable vectors. The model introduces an innovative object-aware prompting strategy that significantly enhances its capability in handling long-tail data. Experiments on the standard COCO dataset demonstrate that ViTOC outperforms baseline models across all evaluation metrics. Additionally, we propose a reference-free evaluation method based on CLIP to further validate the model's effectiveness. By utilizing pretrained visual model parameters, ViTOC achieves efficient end-to-end training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。