用语言语义桥接多图像域,提升模型跨域泛化能力
Vision and Language Integration for Domain Generalization
- 以语言语义为桥梁,融合视觉与语言空间
- 通过词向量距离捕捉类别间语义关系,低秩逼近提取共性特征
- 适合跨域场景下需要强泛化能力的研究者
领域泛化旨在通过源域训练构建域不变特征空间,使模型在未知目标域上仍具备鲁棒性能。然而由于域间差异,难以找到可靠的共通图像特征空间,根源在于图像缺乏合适的语义基本单元。不同于视觉空间中的图像,语言具有丰富的表达元素,可有效传递语义。受语言语义完整性和图像直观性的启发,我们提出VLCA方法,将语言空间与视觉空间结合,利用语义空间作为桥梁连接多个图像域。具体地,在语言空间中,借助语言基本单元的完整性,通过词向量距离捕捉类别间的语义关系;在视觉空间中,利用图像特征的直观性,通过低秩近似探索同类别样本的共性模式。最终,通过图文多模态空间对齐语言表示与视觉表示。实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Domain generalization aims at training on source domains to uncover a domain-invariant feature space, allowing the model to perform robust generalization ability on unknown target domains. However, due to domain gaps, it is hard to find reliable common image feature space, and the reason for that is the lack of suitable basic units for images. Different from image in vision space, language has comprehensive expression elements that can effectively convey semantics. Inspired by the semantic completeness of language and intuitiveness of image, we propose VLCA, which combine language space and vision space, and connect the multiple image domains by using semantic space as the bridge domain. Specifically, in language space, by taking advantage of the completeness of language basic units, we tend to capture the semantic representation of the relations between categories through word vector distance. Then, in vision space, by taking advantage of the intuitiveness of image features, the common pattern of sample features with the same class is explored through low-rank approximation. In the end, the language representation is aligned with the vision representation through the multimodal space of text and image. Experiments demonstrate the effectiveness of the proposed method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。