无需训练,动态优化文本嵌入以应对测试时分布偏移
Reward-Guided Semantic Evolution for Test-time Adaptive Object Detection

- 测试时通过进化搜索思想调整文本嵌入,不依赖反向传播
- 在多个基准上达到最优性能,计算开销极小
- 适合部署在视觉语言模型的开放词汇目标检测场景
基于视觉语言模型(如 Grounding DINO)的开放词汇目标检测在测试时分布偏移下性能下降,主要源于文本嵌入与区域提议视觉嵌入之间的语义错位。现有测试时自适应方法或依赖昂贵的反向传播,或通过外部记忆绕过错位问题,但均未直接高效地实现文本与视觉的对齐。为此,我们提出无需训练的奖励引导语义演化(RGSE)框架,直接在测试时优化文本嵌入。受进化搜索启发,RGSE将文本嵌入适配视为语义搜索过程:扰动文本嵌入生成候选变体,以当前及历史高置信度视觉提议的余弦相似度作为奖励信号评估,再通过奖励加权平均融合为精化嵌入。无需反向传播,RGSE在多个检测基准上实现最先进性能,同时仅引入极小计算开销。代码将在发表后开源。
原文摘要 · Abstract (English)
Open-vocabulary object detection with vision-language models (VLMs) such as Grounding DINO suffers from performance degradation under test-time distribution shifts, primarily due to semantic misalignment between text embeddings and shifted visual embeddings of region proposals. While recent test-time adaptive object detection methods for VLM-based either rely on costly backpropagation or bypass semantic misalignment via external memory, none directly and efficiently align text and vision in a training-free manner. To address this, we propose Reward-Guided Semantic Evolution (RGSE), a training-free framework that directly refines the text embeddings at test time. Inspired by evolutionary search, RGSE treats text embedding adaptation as a semantic search process: it perturbs text embeddings as candidate variants, evaluates them via cosine similarity with current and historical high-confidence visual proposals as a reward signal, and fuses them into a refined embedding through reward-weighted averaging. Without any backpropagation, RGSE achieves state-of-the-art performance across multiple detection benchmarks while adding minimal computational overhead. Our code will be open source upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。