通过自适应语言引导定位关键区域,提升零样本图像文本对齐效率与准确率。
LAGO: Language-Guided Adaptive Object-Region Focus for Zero-Shot Visual-Text Alignment

- 先无类别地发现候选物体区域,再用语言引导逐步优化定位。
- 在多个基准上超越现有方法,推理时仅需少量候选区域。
- 避免早期误判导致的错误循环,适合细粒度零样本识别场景。
零样本识别旨在无需任务特定监督的情况下,从候选类别描述中选择最匹配的标签。在细粒度场景中,相关证据常存在于局部区域、属性或纹理而非整图,因此全图对齐效果不佳。现有局部对齐方法虽通过多区域比对改进性能,但通常依赖大量随机或冗余裁剪,增加推理开销并引入大量弱相关候选。此外,过早引入语义指导可能引发错误放大反馈——错误中间预测影响后续定位并强化错误,称为预测环。本文提出LAGO(语言引导自适应物体区域聚焦),一种高效且鲁棒的零样本局部视觉-文本对齐框架。LAGO首先进行无类别物体中心候选发现以获得稳定视觉初始化,随后采用受中间置信度控制的语言引导自适应精炼。同时通过物体-上下文双通道聚合策略融合物体级、上下文及整图证据。大量实验表明,LAGO在标准零样本基准和挑战性分布偏移设置下持续达到领先性能,且推理时所需候选区域显著减少。
原文摘要 · Abstract (English)
Zero-shot recognition aims to classify an image by selecting the most compatible label description from a set of candidate classes without any task-specific supervision. In fine-grained settings, however, the relevant evidence often lies in localized parts, attributes, or textures rather than in the full image, making whole-image alignment suboptimal. Recent localized visual-text alignment methods address this by comparing class descriptions with multiple image regions, but they typically rely on large sets of random or redundant crops, increasing inference cost and introducing many highly redundant or weakly relevant candidates. Moreover, introducing semantic guidance too early can create an error-amplifying feedback process in which inaccurate intermediate predictions bias later localization and reinforce subsequent mistakes; we refer to this failure mode as the prediction loop. We propose LAGO (LAnguage-Guided adaptive Object-region focus), a framework for efficient and robust zero-shot localized visual-text alignment. LAGO first performs class-agnostic object-centric candidate discovery to obtain a stable visual initialization, and then applies adaptive language-guided refinement with the strength of semantic guidance controlled by intermediate confidence. It further combines object-level, contextual, and full-image evidence through an effective object-context dual-channel aggregation strategy. Extensive experiments show that LAGO consistently achieves state-of-the-art performance on standard zero-shot benchmarks and challenging distribution-shift settings, while requiring substantially fewer candidate regions at inference time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。