通过解耦对齐机制,显著提升视觉语言模型的跨模态兼容性。
The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

- 分解模态差距为中心偏移与分布不一致,精准定位问题根源。
- 三阶段课程训练使跨模态差距减少82.3%,图文聚类准确率提升至0.516。
- 适用于需要高精度跨模态交互的任务,如图像描述与联合聚类。
视觉语言模型(如CLIP)虽在图像与文本间建立共享嵌入空间,但其表示仍存在几何分离现象,即模态间隙,限制了需跨模态互换性的任务(如图像描述、联合聚类)。现有后处理方法仅缓解全局中心偏移,未解决分布不一致问题。本文将模态间隙分解为中心距差与分布距差,实验证明分布距差是跨模态任务质量的强预测因子(R²=0.986),而常用原始差距则误导(R²=0.691)。为此提出三阶段课程对齐框架TPC-CMA,通过联合校正中心偏移与重塑分布结构,并采用梯度感知调度实现稳定优化。实验表明,当α_target=0.05时,模态间隙降低66.6%,准确率仅下降4.84%;当α_target=0.5时,间隙减少82.3%,聚类ARI从0.318升至0.516,图像描述指标CIDEr提升57.1%。代码与预训练模型将在接受后公开。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) such as CLIP learn a shared embedding space for images and text, yet their representations remain geometrically separated, a phenomenon known as the modality gap. This gap limits tasks requiring cross-modal interchangeability, such as captioning and joint clustering. Existing post-processing approaches can partially improve cross-modal compatibility; however, we show through geometric analysis that they primarily reduce the global centroid offset while leaving the underlying distributional mismatch intact. We decompose the modality gap into a Centroid Gap and a Distribution Gap, and demonstrate that the Distribution Gap is the true predictor of cross-modal task quality ($R^2 = 0.986$), whereas the commonly used Raw Gap is misleading ($R^2 = 0.691$). Motivated by this observation, we propose TPC-CMA (Three-Phase Curriculum for Cross-Modal Alignment), a fine-tuning framework that explicitly reduces both components. The proposed CMA jointly mitigates centroid offsets and reshapes the distributional structure, while a three-phase curriculum with gradient-aware scheduling progressively introduces alignment during training to enable stable optimization. Experiments demonstrate that our method significantly improves cross-modal alignment. With $α_{\text{target}}{=}0.05$, the modality gap is reduced by 66.6\% with only 4.84\% accuracy drop. Under stronger alignment ($α_{\text{target}}{=}0.5$), the gap is reduced by 82.3\%, clustering ARI improves from 0.318 to 0.516, and captioning CIDEr increases by 57.1\% over the original model. Our code and pre-trained models will be made publicly available upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。