无需微调，动态匹配适配器实现开放词汇语义分割的测试时域自适应。
Semantic Library Adaptation: LoRA Retrieval and Fusion for Open-Vocabulary Semantic Segmentation
- 用CLIP嵌入索引LoRA适配器库，按输入域相似度动态融合
- 在20个域的基准上超越现有方法，性能稳定提升5.3%平均mIoU
- 适合隐私敏感场景，可解释性强且无需训练
开放词汇语义分割模型通过视觉与文本关联，对未定义类别进行像素级标注，适用于新数据集。但训练与测试域差异大时性能下降，通常需微调。本文提出语义库自适应（SemLA），一种无需训练的测试时域自适应框架。该方法利用基于LoRA的适配器库，以CLIP嵌入为索引，根据目标域在嵌入空间中的距离，动态融合最相关的适配器，为每个输入构建临时模型。该方法高效扩展，可追踪适配器贡献增强可解释性，并天然保护数据隐私，适用于敏感应用。在覆盖10个标准数据集的20域基准上全面实验表明，SemLA在多种设置下展现出优越的适应能力与性能，确立了开放词汇语义分割域自适应的新标准。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation models associate vision and text to label pixels from an undefined set of classes using textual queries, providing versatile performance on novel datasets. However, large shifts between training and test domains degrade their performance, requiring fine-tuning for effective real-world applications. We introduce Semantic Library Adaptation (SemLA), a novel framework for training-free, test-time domain adaptation. SemLA leverages a library of LoRA-based adapters indexed with CLIP embeddings, dynamically merging the most relevant adapters based on proximity to the target domain in the embedding space. This approach constructs an ad-hoc model tailored to each specific input without additional training. Our method scales efficiently, enhances explainability by tracking adapter contributions, and inherently protects data privacy, making it ideal for sensitive applications. Comprehensive experiments on a 20-domain benchmark built over 10 standard datasets demonstrate SemLA's superior adaptability and performance across diverse settings, establishing a new standard in domain adaptation for open-vocabulary semantic segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。