通过融合区域与分割特征,提升图像描述的准确性与相关性。
Dual-Stream Collaborative Transformer for Image Captioning
- 设计双流协同架构,动态融合区域与分割特征
- 在COCO和Flickr30k上显著超越现有模型
- 适合需要高精度图像描述的应用场景
当前基于区域特征的图像描述方法进展迅速,性能优异,但仍因缺乏上下文信息且过度依赖部分生成内容,易产生无关描述。本文提出双流协同变换器(DSCT),引入分割特征以解决该问题。DSCT通过整合并融合区域与分割特征,指导句子生成。其包含多个模式特定互注意力编码器(PSMAE)和动态提名解码器(DND)。PSMAE通过相互查询有效突出并整合两种表示的私有信息;DND动态选择最相关的学习模块,并利用整合后区域与分割特征间的同质特性,生成更准确、更丰富的描述。据我们所知,这是首次探索如何以动态方式融合不同模式特性的研究,以规避语义不一致与空间错位问题。在主流基准数据集上的实验结果表明,DSCT优于现有先进图像描述模型。
原文摘要 · Abstract (English)
Current region feature-based image captioning methods have progressed rapidly and achieved remarkable performance. However, they are still prone to generating irrelevant descriptions due to the lack of contextual information and the over-reliance on generated partial descriptions for predicting the remaining words. In this paper, we propose a Dual-Stream Collaborative Transformer (DSCT) to address this issue by introducing the segmentation feature. The proposed DSCT consolidates and then fuses the region and segmentation features to guide the generation of caption sentences. It contains multiple Pattern-Specific Mutual Attention Encoders (PSMAEs) and Dynamic Nomination Decoders (DNDs). The PSMAE effectively highlights and consolidates the private information of two representations by querying each other. The DND dynamically searches for the most relevant learning blocks to the input textual representations and exploits the homogeneous features between the consolidated region and segmentation features to generate more accurate and descriptive caption sentences. To the best of our knowledge, this is the first study to explore how to fuse different pattern-specific features in a dynamic way to bypass their semantic inconsistencies and spatial misalignment issues for image captioning. The experimental results from popular benchmark datasets demonstrate that our DSCT outperforms the state-of-the-art image captioning models in the literature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。