arXiv:2604.24642cs.CV2026-04

探究CLIP对全景图文语义的理解能力,发现其文本理解强但视觉不变性弱。

Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics

论文配图:Probing CLIP's Comprehension of 360-Degree Textual and Visual Semantics
图 1 · 摘自论文原文
  • 用关键词操控和水平环移测试CLIP对全景语义的把握
  • 能理解显式文本标识,但对环移后语义保持差
  • 提出LoRA微调增强环移不变性,有性能权衡

从文本快速生成丰富全景世界正成为现实,但其语义对齐的可靠评估仍存空白。主流的对比语言-图像预训练(CLIP)模型主要基于视角图像-文本对训练,其对360度全景图像-文本对独特特性的理解尚不明确。本文提出两个概念:360度文本语义(由显式格式标识传递)与360度视觉语义(在水平环移下保持不变)。通过关键词操纵和不同幅度的水平环移,对多个CLIP配置进行严格统计分析,发现:(1) CLIP能有效利用显式文本标识,理解360度文本语义;(2) 在水平环移下语义对齐不鲁棒,表明对360度视觉语义理解有限。为此,提出基于LoRA的微调框架,显式引入环移不变性。微调后模型对360度视觉语义理解提升,但原语义评估性能略有下降,揭示适应全景图时的根本权衡。代码已开源:https://github.com/littlewhitesea/360Semantics。

原文摘要 · Abstract (English)

The dream of instantly creating rich 360-degree panoramic worlds from text is rapidly becoming a reality, yet a crucial gap exists in our ability to reliably evaluate their semantic alignment. Contrastive Language-Image Pre-training (CLIP) models, standard AI evaluators, predominantly trained on perspective image-text pairs, face an open question regarding their understanding of the unique characteristics of 360-degree panoramic image-text pairs. This paper addresses this gap by first introducing two concepts: \emph{360-degree textual semantics}, semantic information conveyed by explicit format identifiers, and \emph{360-degree visual semantics}, invariant semantics under horizontal circular shifts. To probe CLIP's comprehension of these semantics, we then propose novel evaluation methodologies using keyword manipulation and horizontal circular shifts of varying magnitudes. Rigorous statistical analyses across popular CLIP configurations reveal that: (1) CLIP models effectively leverage explicit textual identifiers, demonstrating an understanding of 360-degree textual semantics; and (2) CLIP models fail to robustly preserve semantic alignment under horizontal circular shifts, indicating limited comprehension of 360-degree visual semantics. To address this limitation, we propose a LoRA-based fine-tuning framework that explicitly instills invariance to circular shifts. Our fine-tuned models exhibit improved comprehension of 360-degree visual semantics, though with a slight degradation in original semantic evaluation performance, highlighting a fundamental trade-off in adapting CLIP to 360-degree panoramic images. Code is available at https://github.com/littlewhitesea/360Semantics.

CLIP全景图像语义对齐微调

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。