提升多语言图文模型的跨模态对齐，增强语义结构一致性。
Topological Alignment of Shared Vision-Language Embedding Space
- 用拓扑保持约束优化多语言嵌入空间结构
- 在CIFAR-100上零样本准确率提升，xFlickr&CO检索性能更强
- 适用于多语言视觉语言模型，也可推广至其他表示学习
对比式视觉语言模型（VLMs）展现出强大的零样本能力，但其跨模态对齐仍偏向英语，因多语言多模态数据有限。近期的多语言扩展虽缓解了这一差距，但仅强制实例级对齐，忽视共享嵌入空间的全局几何结构。本文提出ToMCLIP（Topological Alignment for Multilingual CLIP），一种拓扑感知框架，通过拓扑保持约束对齐嵌入空间。方法利用持久同调定义拓扑对齐损失，并通过图稀疏化策略近似持久图谱，给出理论误差界。实验验证该方法提升了多语言表示的结构一致性，在CIFAR-100上实现更高零样本准确率，在xFlickr&CO上取得更强多语言检索性能。该方法不仅适用于VLMs，还可作为通用技术融入各类表示学习任务。代码已公开于https://github.com/junwon0/ToMCLIP.git。
原文摘要 · Abstract (English)
Contrastive Vision-Language Models (VLMs) have demonstrated strong zero-shot capabilities. However, their cross-modal alignment remains biased toward English due to limited multilingual multimodal data. Recent multilingual extensions have alleviated this gap but enforce instance-level alignment while neglecting the global geometry of the shared embedding space. We address this problem by introducing ToMCLIP (Topological Alignment for Multilingual CLIP), a topology-aware framework aligning embedding spaces with topology-preserving constraints. The proposed method applies persistent homology to define a topological alignment loss and approximates persistence diagram with theoretical error bounds using graph sparsification strategy. This work validates the proposed approach, showing enhanced structural coherence of multilingual representations, higher zero-shot accuracy on the CIFAR-100, and stronger multilingual retrieval performance on the xFlickr&CO. Beyond VLMs, the proposed approach provides a general method for incorporating topological alignment into representation learning. Code is available at https://github.com/junwon0/ToMCLIP.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。