用视觉信息帮大模型分辨词语歧义，效果显著提升。
Visual Word Sense Disambiguation with CLIP through Dual-Channel Text Prompting and Image Augmentations
- 双通道文本提示融合语义与图像特征，增强语义表达
- 在SemEval-2023数据集上MRR达0.7590，命中率62.20%
- 精准对齐CLIP的提示设计比复杂增广更有效
词汇歧义是大语言模型理解自然语言的持续挑战。本文提出一种可解释的视觉词义消歧（VWSD）框架，利用CLIP将含糊语言与候选图像映射到共享多模态空间。通过结合WordNet同义词的语义提示与基于图像的提示构成双通道文本嵌入，并采用鲁棒的测试时增强策略优化图像嵌入。最终通过余弦相似度匹配最契合模糊文本的图像。在SemEval-2023 VWSD数据集上的实验表明，嵌入增强使MRR从0.7227提升至0.7590，命中率从0.5810提升至0.6220。消融实验证明，双通道提示带来强且低延迟性能，而激进图像增强仅带来微弱收益。额外实验显示，使用WordNet定义或跨语言提示集合会引入噪声，削弱语义特异性，凸显精确对齐CLIP提示在视觉词义消歧中的有效性。
原文摘要 · Abstract (English)
Ambiguity poses persistent challenges in natural language understanding for large language models (LLMs). To better understand how lexical ambiguity can be resolved through the visual domain, we develop an interpretable Visual Word Sense Disambiguation (VWSD) framework. The model leverages CLIP to project ambiguous language and candidate images into a shared multimodal space. We enrich textual embeddings using a dual-channel ensemble of semantic and photo-based prompts with WordNet synonyms, while image embeddings are refined through robust test-time augmentations. We then use cosine similarity to determine the image that best aligns with the ambiguous text. When evaluated on the SemEval-2023 VWSD dataset, enriching the embeddings raises the MRR from 0.7227 to 0.7590 and the Hit Rate from 0.5810 to 0.6220. Ablation studies reveal that dual-channel prompting provides strong, low-latency performance, whereas aggressive image augmentation yields only marginal gains. Additional experiments with WordNet definitions and multilingual prompt ensembles further suggest that noisy external signals tend to dilute semantic specificity, reinforcing the effectiveness of precise, CLIP-aligned prompts for visual word sense disambiguation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。