让视觉语言模型在未知概念上表现更好,通过动态生成语义词嵌入解决跨模态对齐崩溃问题。
GTMA: Dynamic Representation Optimization for OOD Vision-Language Models
- 推理时动态生成连续伪词嵌入,突破固定词汇表限制。
- 在ImageNet-R和VISTA-Beyond上零样本性能提升15-20%。
- 适合需要强泛化能力的开放世界视觉语言应用。
视觉语言模型(VLMs)在开放世界应用中面临分布外(OOD)概念时,常因跨模态对齐崩溃而导致零样本性能严重下降。我们发现根源在于模态不对称:尽管视觉编码器能从未见图像中提取判别特征,但文本编码器受限于固定离散词汇表,无法生成新的语义锚点。现有方法如CoOp或LoRA仅部分缓解此问题,仍局限于预训练语义空间。为此,我们提出动态表示优化,通过引导目标匹配适应(GTMA)框架实现。推理时,GTMA构建与未知图像视觉锚点最佳对齐的连续伪词嵌入,有效绕过词汇限制。优化由自适应基于梯度的表示策略算法驱动,并引入语义正则化以保持合理性及与模型先验知识的兼容性。在ImageNet-R和VISTA-Beyond基准上的实验表明,GTMA相比基线VLM在零样本和少样本OOD准确率上提升15-20%,同时保持对分布内概念的性能。消融实验进一步证实伪词优化的必要性。
原文摘要 · Abstract (English)
Vision-language models (VLMs) struggle in open-world applications, where out-of-distribution (OOD) concepts often trigger cross-modal alignment collapse and severely degrade zero-shot performance. We identify the root cause as modal asymmetry: while the visual encoder can extract discriminative features from unseen images, the text encoder is constrained by a fixed discrete vocabulary and cannot synthesize new semantic anchors. Existing approaches such as CoOp or LoRA provide only partial remedies, as they remain confined to the pre-trained semantic space. To overcome this bottleneck, we propose dynamic representation optimization, realized through the Guided Target-Matching Adaptation (GTMA) framework. At inference time, GTMA constructs a continuous pseudo-word embedding that best aligns with an OOD image's visual anchor, effectively bypassing vocabulary limitations. The optimization is driven by an adaptive gradient-based representation policy optimization algorithm, which incorporates semantic regularization to preserve plausibility and compatibility with the model's prior knowledge. Experiments on ImageNet-R and the VISTA-Beyond benchmark demonstrate that GTMA improves zero-shot and few-shot OOD accuracy by up to 15-20 percent over the base VLM while maintaining performance on in-distribution concepts. Ablation studies further confirm the necessity of pseudo-word optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。