arXiv:2604.00279cs.CVcs.AI2026-04

通过解耦对齐机制,显著提升视觉语言模型的跨模态兼容性。

The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment

论文配图:The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
图 1 · 摘自论文原文
  • 分解模态差距为中心偏移与分布不一致,精准定位问题根源。
  • 三阶段课程训练使跨模态差距减少82.3%,图文聚类准确率提升至0.516。
  • 适用于需要高精度跨模态交互的任务,如图像描述与联合聚类。

视觉语言模型(如CLIP)虽在图像与文本间建立共享嵌入空间,但其表示仍存在几何分离现象,即模态间隙,限制了需跨模态互换性的任务(如图像描述、联合聚类)。现有后处理方法仅缓解全局中心偏移,未解决分布不一致问题。本文将模态间隙分解为中心距差与分布距差,实验证明分布距差是跨模态任务质量的强预测因子(R²=0.986),而常用原始差距则误导(R²=0.691)。为此提出三阶段课程对齐框架TPC-CMA,通过联合校正中心偏移与重塑分布结构,并采用梯度感知调度实现稳定优化。实验表明,当α_target=0.05时,模态间隙降低66.6%,准确率仅下降4.84%;当α_target=0.5时,间隙减少82.3%,聚类ARI从0.318升至0.516,图像描述指标CIDEr提升57.1%。代码与预训练模型将在接受后公开。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) such as CLIP learn a shared embedding space for images and text, yet their representations remain geometrically separated, a phenomenon known as the modality gap. This gap limits tasks requiring cross-modal interchangeability, such as captioning and joint clustering. Existing post-processing approaches can partially improve cross-modal compatibility; however, we show through geometric analysis that they primarily reduce the global centroid offset while leaving the underlying distributional mismatch intact. We decompose the modality gap into a Centroid Gap and a Distribution Gap, and demonstrate that the Distribution Gap is the true predictor of cross-modal task quality ($R^2 = 0.986$), whereas the commonly used Raw Gap is misleading ($R^2 = 0.691$). Motivated by this observation, we propose TPC-CMA (Three-Phase Curriculum for Cross-Modal Alignment), a fine-tuning framework that explicitly reduces both components. The proposed CMA jointly mitigates centroid offsets and reshapes the distributional structure, while a three-phase curriculum with gradient-aware scheduling progressively introduces alignment during training to enable stable optimization. Experiments demonstrate that our method significantly improves cross-modal alignment. With $α_{\text{target}}{=}0.05$, the modality gap is reduced by 66.6\% with only 4.84\% accuracy drop. Under stronger alignment ($α_{\text{target}}{=}0.5$), the gap is reduced by 82.3\%, clustering ARI improves from 0.318 to 0.516, and captioning CIDEr increases by 57.1\% over the original model. Our code and pre-trained models will be made publicly available upon acceptance.

视觉语言模型跨模态对齐生成能力结构优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。