提出DAF框架，让视觉语言模型在持续分布变化下保持稳定分割能力。
Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation
- 引入多样性损失、跨模态锚点一致性和特征显著性过滤，抑制特征崩溃
- 在五个数据集上相比源模型提升超8点mIoU，抗干扰能力更强
- 适合需要长期稳定运行的开放词汇语义分割场景
开放词汇语义分割依赖视觉-语言对齐识别任意文本定义类别，但在持续测试时分布漂移下该对齐易失效。诊断分析发现：熵最小化导致局部类坍塌，持续更新破坏对齐，低漂移样本的冗余梯度浪费计算。本文提出DAF框架，通过边际多样性损失抵抗坍塌，跨模态锚点一致性损失约束特征相对于冻结源模型的漂移，并以特征显著性过滤跳过低价值反向传播，缓解源锚开销。在涵盖自然场景、自动驾驶、水下图像和遥感的五个数据集及其损坏版本上评估。在连续分布漂移下，DAF保持稳定，而熵最小化会崩溃；相比源模型，其在Pascal VOC20-C上提升超8点mIoU，LoveDA上超9点，Foggy Cityscapes上超3点，且对激进适配和学习率选择鲁棒。
原文摘要 · Abstract (English)
Open-vocabulary semantic segmentation (OVSS) relies on vision-language alignment to recognize arbitrary text-defined categories, yet this alignment is fragile under continual test-time distribution shift. Our diagnostic analysis reveals that entropy minimization drives patch-level class collapse, continual updates erode vision-language alignment, and redundant gradients from low-shift samples waste computation. We propose Diversify, Anchor, and Filter (DAF), a stabilization framework that augments entropy-based adaptation with a marginal diversity loss that resists collapse, a cross-modal anchor consistency loss that constrains feature drift relative to a frozen source model, and feature salience filtering that skips low-value backward passes to offset part of the source-anchor overhead. We evaluate on five datasets spanning natural scenes, autonomous driving, underwater imagery, and remote sensing with their corrupted variants. Across the evaluated continual shifts, DAF remains stable where entropy minimization collapses, improving mIoU by over 8 points on Pascal VOC20-C, over 9 points on LoveDA, and over 3 points on Foggy Cityscapes compared to the source model, and is robust to aggressive adaptation and learning rate choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。