微小语义变化让视觉语言模型选错目标，稳定性堪忧
Prompt Sensitivity in Vision-Language Grounding: How Small Changes in Wording Affect Object Detection
- 用六组近义词测试，发现同一图像下选择对象差异大
- 平均每次查询选出不同物体2.11次，且方向性明显
- 文字相似度只解释34%偏差，根本问题在选择机制
视觉语言模型通过自然语言查询实现开放词汇的目标定位，隐含假设是语义等价的描述应产生一致结果。我们使用包含DETR生成候选框与CLIP进行语言条件筛选的可控流程，在263张COCO val2017图像上检验该假设。发现如“a person”、“a human”和“a pedestrian”等重叠提示常选出不同实例，六组提示间平均不稳定性达2.11个不同选择。主成分分析显示这种变异具有结构化和方向性，非随机。提示集成并未提升质量，反而常将选择导向通用区域。进一步表明，文本嵌入距离仅能解释34%的定位分歧（r = -0.58），证实不稳定性源于argmax选择机制，而非单纯文本距离。
原文摘要 · Abstract (English)
Vision-language models enable open-vocabulary object grounding through natural language queries, under the implicit assumption that semantically equivalent descriptions yield consistent outputs. We examine this assumption using a controlled pipeline combining DETR for object proposals with CLIP for language-conditioned selection on 263 COCO val2017 images. We find that overlapping prompts such as "a person," "a human," and "a pedestrian" frequently select different instances, with mean instability of 2.11 distinct selections across six prompts. PCA analysis shows this variability is structured and directional, not random. Prompt ensembling does not improve quality and often shifts selections toward generic regions. We further show that text embedding proximity explains only 34% of grounding disagreement (r = -0.58), confirming that instability arises from the argmax selection mechanism rather than text-level distances alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。