提出三阶段对齐网络,提升复杂场景下图像与文本的精准匹配能力。
TFANet: Three-Stage Image-Text Feature Alignment Network for Robust Referring Image Segmentation
- 分三阶段逐步增强图像与文本特征对齐,从多尺度交互到跨模态扫描再到语义深化。
- 在 RefCOCO+/RefCOCO++/RefCLEF 数据集上平均 mAP 达 63.7%,优于现有方法。
- 适合处理包含相似物体的复杂图像分割任务,尤其关注语言描述精确性。
指代图像分割(RIS)旨在根据语言表达分割图像区域,需在多模态间实现细粒度对齐。现有方法常因多模态错位和语言语义损失而表现不佳,尤其在含多个视觉相似对象的复杂场景中,目标常被误定位或分割不完整。为此,本文提出 TFANet——一种三阶段图像-文本特征对齐网络,通过层级化框架系统性提升多模态对齐:第一阶段设计多尺度线性交叉注意力模块(MLAM),在多尺度上促进视觉特征与文本表示间的双向语义交互,建立图像区域与不同粒度语言描述之间的丰富高效对齐;第二阶段通过跨模态特征扫描模块(CFSM)实施多模态选择性扫描,捕捉长程依赖并构建统一的多模态表征,有效建模长程跨模态依赖,提升复杂场景中的对齐精度;第三阶段提出词级语言特征引导的语义深化模块(WFDM),补偿前序阶段引入的语义退化问题。
原文摘要 · Abstract (English)
Referring Image Segmentation (RIS) is a task that segments image regions based on language expressions, requiring fine-grained alignment between two modalities. However, existing methods often struggle with multimodal misalignment and language semantic loss, especially in complex scenes containing multiple visually similar objects, where uniquely described targets are frequently mislocalized or incompletely segmented. To tackle these challenges, this paper proposes TFANet, a Three-stage Image-Text Feature Alignment Network that systematically enhances multimodal alignment through a hierarchical framework comprising three stages: Knowledge Plus Stage (KPS), Knowledge Fusion Stage (KFS), and Knowledge Intensification Stage (KIS). In the first stage, we design the Multiscale Linear Cross-Attention Module (MLAM), which facilitates bidirectional semantic exchange between visual features and textual representations across multiple scales. This establishes rich and efficient alignment between image regions and different granularities of linguistic descriptions. Subsequently, the KFS further strengthens feature alignment through the Cross-modal Feature Scanning Module (CFSM), which applies multimodal selective scanning to capture long-range dependencies and construct a unified multimodal representation. This is essential for modeling long-range cross-modal dependencies and enhancing alignment accuracy in complex scenes. Finally, in the KIS, we propose the Word-level Linguistic Feature-guided Semantic Deepening Module (WFDM) to compensate for semantic degradation introduced in earlier stages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。