优化视觉语言模型，让其更依赖图像而非文字描述。
Cross-Modal Attention Analysis and Optimization in Vision-Language Models: A Study on Visual Reliability
- 用对抗样本测试模型对图文依赖程度，量化文本捷径问题。
- 优化后模型在冲突图文下准确率下降仅9.8%，提升64.4%。
- 适合关注多模态模型可靠性与对齐效果的研究者。
视觉语言模型虽表现优异，但存在过度依赖文本、忽视视觉信息的“文本捷径学习”问题。本文提出一种对抗评估框架，通过将语义冲突的文本与不变图像配对，测量准确率下降（Drop）来量化跨模态依赖。在包含1000张几何图形的可控数据集上，采用四种对抗策略（形状互换、颜色互换、位置互换、随机文本）。对比三种配置：基线CLIP（ViT-B/32）、LoRA微调及集成硬负样本挖掘、标签平滑、分层学习率、余弦重启、课程学习与数据增强的优化版LoRA。结果显示，优化模型将平均Drop从27.5%降至9.8%（相对提升64.4%，p<0.001），同时保持97%正常准确率。注意力可视化与嵌入空间分析表明，优化模型更关注视觉特征，实现更紧密的跨模态对齐。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) achieve strong cross-modal performance, yet recent evidence suggests they over-rely on textual descriptions while under-utilizing visual evidence -- a phenomenon termed ``text shortcut learning.'' We propose an adversarial evaluation framework that quantifies this cross-modal dependency by measuring accuracy degradation (Drop) when semantically conflicting text is paired with unchanged images. Four adversarial strategies -- shape\_swap, color\_swap, position\_swap, and random\_text -- are applied to a controlled geometric-shapes dataset ($n{=}1{,}000$). We compare three configurations: Baseline CLIP (ViT-B/32), LoRA fine-tuning, and LoRA Optimized (integrating Hard Negative Mining, Label Smoothing, layer-wise learning rates, Cosine Restarts, curriculum learning, and data augmentation). The optimized model reduces average Drop from 27.5\% to 9.8\% (64.4\% relative improvement, $p{<}0.001$) while maintaining 97\% normal accuracy. Attention visualization and embedding-space analysis confirm that the optimized model attends more to visual features and achieves tighter cross-modal alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。