让视觉语言模型正确理解逻辑组合查询，解决'既...又不...'的误检问题。
Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models
- 分离概念提取与逻辑约束执行，用外部方法处理逻辑运算
- 在FACTOR-Bench上达到85.5%准确率，显著超越微调模型
- 无需训练即可提升复杂查询准确率，适合需要精准语义理解的场景
双编码器视觉语言模型通过相似度接口实现零样本检索，但无法满足组合约束，例如'雨伞且无人'的查询仍会返回含人的图像，尽管概念检测可靠。我们发现这是接口层面的'概念词袋效应'所致：相似度分数近似于概念证据的均值聚合，无视逻辑运算符。尽管文本嵌入中存在运算符相关信号，但其强度不足或对齐错误，难以影响排序。微调无法可靠解决此问题，因瓶颈在于相似度如何聚合证据，而非编码器表征内容。我们提出因子推理（factored inference），将证据提取与约束执行分离，并引入无需训练的LCSE（逻辑约束得分编辑）方法，利用冻结编码器输出的概念得分在外部执行约束。我们还构建了FACTOR-Bench基准，结果显示LCSE在该基准上达到85.5%准确率，优于最佳微调基线的73.2%，在SigLIP 2上达90.7%；同时将NegBench COCO MCQ准确率从27.2%提升至65.2%，并保持原有检索性能。
原文摘要 · Abstract (English)
Dual-encoder vision-language models (VLMs) expose a similarity interface that enables zero-shot retrieval but fails compositional constraints: queries like "umbrella and no person" retrieve images containing both, even when concept detection is reliable. We trace this to an interface-level Bag-of-Concepts effect, where similarity scores approximate mean pooling of concept evidence regardless of operators. Although operator-dependent signals exist in text embeddings, they are too weak or misaligned to affect rankings. Fine-tuning does not reliably resolve this failure because the dominant bottleneck is how similarity aggregates evidence rather than what encoders represent. We propose factored inference, which separates evidence extraction from constraint execution, and introduce LCSE (Logic-Constrained Score Editing), a training-free method that executes constraints externally using concept scores from frozen encoders. We also introduce FACTOR-Bench, where LCSE achieves 85.5% accuracy versus 73.2% for the best fine-tuned baseline, 90.7% when applied to SigLIP 2, and improves NegBench COCO MCQ accuracy from 27.2% to 65.2% while preserving retrieval performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。