让图像和文本的层次特征在双曲空间中对齐,提升跨模态理解效果。
Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
- 为图像和文本构建树状层次特征,用交叉注意力实现语义引导的特征提取。
- 在不同曲率的双曲空间中嵌入特征树,并通过中间流形实现跨空间对齐。
- 在少样本与跨域任务中显著优于基线,适合需要层次化理解的多模态场景。
跨模态对齐对视觉语言模型有效融合多源信息至关重要。现有方法通常从文本中提取层次特征,而将每张图像表示为单一特征,导致对齐不对称且效果不佳。为此,我们提出「跨树对齐」(Alignment across Trees),为图像和文本模态分别构建并对其树状层次特征。具体地,设计了一种语义感知的视觉特征提取框架,利用中间Transformer层的视觉类别令牌,通过文本提示引导的交叉注意力机制,实现由粗到细的语义特征提取。随后,将两种模态的特征树嵌入具有不同曲率的双曲流形中,以有效建模其层次结构。为在异质双曲流形间实现对齐,我们定义了分布在不同流形上的分布间的KL距离,并通过最小化该距离学习一个中间流形进行对齐,证明了最优中间流形的存在性与唯一性。在多个图像数据集上的分类任务实验表明,该方法在少样本与跨域设置下均持续优于强基线。
原文摘要 · Abstract (English)
Modality alignment is critical for vision-language models (VLMs) to effectively integrate information across modalities. However, existing methods extract hierarchical features from text while representing each image with a single feature, leading to asymmetric and suboptimal alignment. To address this, we propose Alignment across Trees, a method that constructs and aligns tree-like hierarchical features for both image and text modalities. Specifically, we introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers, guided by textual cues to extract visual features with coarse-to-fine semantics. We then embed the feature trees of the two modalities into hyperbolic manifolds with distinct curvatures to effectively model their hierarchical structures. To align across the heterogeneous hyperbolic manifolds with different curvatures, we formulate a KL distance measure between distributions on heterogeneous manifolds, and learn an intermediate manifold for manifold alignment by minimizing the distance. We prove the existence and uniqueness of the optimal intermediate manifold. Experiments on taxonomic open-set classification tasks across multiple image datasets demonstrate that our method consistently outperforms strong baselines under few-shot and cross-domain settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。