arXiv:2506.08645cs.LG2025-06NeurIPS被引 7

用克罗内克积融合多模型特征,提升跨模态与单模态表现

When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product

  • 通过克罗内克积融合两个嵌入的核函数,实现互补特征联合建模
  • 在多个数据集上提升跨模态性能,如CLIP与DINOv2融合后准确率提高3.2%
  • 提出高效近似方法RP-KrossFuse,支持大规模嵌入融合

当前顶尖嵌入模型常捕捉互为补充的判别性特征:例如某图像模型擅长细粒度纹理区分,另一模型则侧重物体结构。受此启发,我们提出一种基于核函数相乘的融合方法,使两个嵌入的判别结构交互,生成的融合表示其核编码了父模型各自识别出的聚类集合的并集。该形式自然适用于成对多模态数据(如图像-文本对),其中各模态专属核的乘积继承了双域结构。数学上,该核乘积由嵌入特征图的克罗内克积实现,形成所提KrossFuse框架。为降低高维克罗内克空间带来的计算开销,进一步设计了基于随机投影的可扩展版本RP-KrossFuse。作为关键应用,该框架有效缩小了跨模态嵌入(如CLIP、BLIP)与单模态专家(如DINOv2、E5)间的性能差距。实验表明,RP-KrossFuse能有效整合这些模型,在保持跨模态对齐的同时提升模态特定性能。

原文摘要 · Abstract (English)

State-of-the-art embeddings often capture distinct yet complementary discriminative features: For instance, one image embedding model may excel at distinguishing fine-grained textures, while another focuses on object-level structure. Motivated by this observation, we propose a principled approach to fuse such complementary representations through kernel multiplication. Multiplying the kernel similarity functions of two embeddings allows their discriminative structures to interact, producing a fused representation whose kernel encodes the union of the clusters identified by each parent embedding. This formulation also provides a natural way to construct joint kernels for paired multi-modal data (e.g., image-text tuples), where the product of modality-specific kernels inherits structure from both domains. We highlight that this kernel product is mathematically realized via the Kronecker product of the embedding feature maps, yielding our proposed KrossFuse framework for embedding fusion. To address the computational cost of the resulting high-dimensional Kronecker space, we further develop RP-KrossFuse, a scalable variant that leverages random projections for efficient approximation. As a key application, we use this framework to bridge the performance gap between cross-modal embeddings (e.g., CLIP, BLIP) and unimodal experts (e.g., DINOv2, E5). Experiments show that RP-KrossFuse effectively integrates these models, enhancing modality-specific performance while preserving cross-modal alignment. The project code is available at https://github.com/yokiwuuu/KrossFuse.

嵌入融合克罗内克积多模态特征集成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。