用大模型生成细粒度语义描述,提升跨域分割准确率
Cross-Domain Semantic Segmentation with Large Language Model-Assisted Descriptor Generation
- 借助大模型生成上下文敏感的细粒度类别描述
- 在ADE20K和COCO-Stuff上实现最高6.1% mIoU提升
- 无需重训练,适合交互式与领域特定分割场景
语义分割在像素级理解视觉场景中至关重要。传统方法虽取得显著进展,但在不同场景和未见物体类别上的泛化能力仍受限。大语言模型(LLM)为连接视觉与文本模态提供了新路径,可深入理解语义关系。本文提出LangSeg,一种基于LLM引导的语义分割方法,利用LLM生成上下文敏感、细粒度的子类描述,并与预训练视觉变压器(ViT)融合,实现优异分割性能而无需大量重训练。我们在两个挑战性数据集ADE20K和COCO-Stuff上评估该方法,结果表明其优于当前最优模型,mIoU最高提升6.1%。此外,通过全面消融实验和人工评估,验证了该方法在真实场景中的有效性。结果表明,LangSeg不仅在语义理解与上下文对齐方面表现卓越,还为语言引导分割任务提供灵活高效框架,拓展了交互式与领域特定分割应用的可能性。
原文摘要 · Abstract (English)
Semantic segmentation plays a crucial role in enabling machines to understand and interpret visual scenes at a pixel level. While traditional segmentation methods have achieved remarkable success, their generalization to diverse scenes and unseen object categories remains limited. Recent advancements in large language models (LLMs) offer a promising avenue for bridging visual and textual modalities, providing a deeper understanding of semantic relationships. In this paper, we propose LangSeg, a novel LLM-guided semantic segmentation method that leverages context-sensitive, fine-grained subclass descriptors generated by LLMs. Our framework integrates these descriptors with a pre-trained Vision Transformer (ViT) to achieve superior segmentation performance without extensive model retraining. We evaluate LangSeg on two challenging datasets, ADE20K and COCO-Stuff, where it outperforms state-of-the-art models, achieving up to a 6.1% improvement in mean Intersection over Union (mIoU). Additionally, we conduct a comprehensive ablation study and human evaluation to validate the effectiveness of our method in real-world scenarios. The results demonstrate that LangSeg not only excels in semantic understanding and contextual alignment but also provides a flexible and efficient framework for language-guided segmentation tasks. This approach opens up new possibilities for interactive and domain-specific segmentation applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。