用文本增强热像与可见光图像分割，提升复杂场景识别准确率
TASeg: Text-aware RGB-T Semantic Segmentation based on Fine-tuning Vision Foundation Models
- 用低秩微调适配视觉大模型，融合文本与多模态图像特征
- 在多个数据集上以更少参数实现更高精度，尤其擅长区分相似类别的分割
- 适合需要高精度语义理解的自动驾驶、安防监控等场景
开放环境中的可靠语义分割对智能系统至关重要，但现有方法仍存在两大问题：1）现有RGB-T分割模型主要依赖低层视觉特征，缺乏高层文本信息，在类别视觉特征相近时难以准确分割；2）尽管SAM在实例分割上表现优异，但其与热成像和文本的融合受限于模态差异和计算效率。为此，我们提出TASeg，基于低秩微调（LoRA）技术适配视觉基础模型的文本感知型RGB-T分割框架。具体地，在图像编码器中设计动态特征融合模块（DFFM），在冻结SAM原始变换块的前提下有效融合多模态特征；同时在掩码解码器中引入CLIP生成的文本嵌入，实现语义对齐，进一步纠正分类误差并提升语义理解准确性。跨多种数据集的实验表明，该方法在挑战性场景下性能优越，且可训练参数更少。
原文摘要 · Abstract (English)
Reliable semantic segmentation of open environments is essential for intelligent systems, yet significant problems remain: 1) Existing RGB-T semantic segmentation models mainly rely on low-level visual features and lack high-level textual information, which struggle with accurate segmentation when categories share similar visual characteristics. 2) While SAM excels in instance-level segmentation, integrating it with thermal images and text is hindered by modality heterogeneity and computational inefficiency. To address these, we propose TASeg, a text-aware RGB-T segmentation framework by using Low-Rank Adaptation (LoRA) fine-tuning technology to adapt vision foundation models. Specifically, we propose a Dynamic Feature Fusion Module (DFFM) in the image encoder, which effectively merges features from multiple visual modalities while freezing SAM's original transformer blocks. Additionally, we incorporate CLIP-generated text embeddings in the mask decoder to enable semantic alignment, which further rectifies the classification error and improves the semantic understanding accuracy. Experimental results across diverse datasets demonstrate that our method achieves superior performance in challenging scenarios with fewer trainable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。