arXiv:2606.01207cs.CVcs.LG2026-06

特征对齐程度决定融合策略优劣,对齐好时拼接胜过交叉注意力。

Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning

论文配图:Feature Alignment Determines Fusion Strategy: A Comparative Study of Cross-Attention and Concatenation in Multimodal Learning
图 1 · 摘自论文原文
  • 基于特征对齐质量判断该用拼接还是交叉注意力。
  • 对齐时拼接比交叉注意力高4.1-5.1个百分点。
  • 适合构建多模态大模型的融合模块设计参考。

多模态融合中选择交叉注意力还是拼接仍依赖经验直觉而非理论指导。本文表明,特征对齐质量是决定融合策略优劣的关键,而非数据规模。在Flickr8k数据集上,使用ResNet18和CLIP ViT-B/32作为骨干网络的控制实验显示,当特征经视觉-语言预训练目标预对齐后,拼接在2048至16384样本的所有规模下均优于交叉注意力4.1-5.1个百分点。理论分析指出:拼接需约O(d_v + d_t)样本学习融合投影,而交叉注意力需O(d_v * d_t)样本学习双线性注意力权重,对512维特征而言,前者所需样本量不到后者的1/256。特征对齐时,两种方法近似误差差距消失,拼接在实际数据规模下始终更具样本效率。对齐退化实验进一步验证:随着对齐度下降,拼接优势从1.3%增至2.8%。研究为多模态系统融合方法选择提供了原则性框架,对多模态大模型设计具有直接意义。

原文摘要 · Abstract (English)

The choice between cross-attention and concatenation for multimodal fusion remains governed by practitioner intuition rather than principled understanding. In this paper, we demonstrate that feature alignment quality, not data scale alone, is the primary determinant of which fusion strategy excels. Through controlled experiments on Flickr8k using two feature extraction backbones (ResNet18 and CLIP ViT-B/32), we show that concatenation outperforms cross-attention by 4.1-5.1 percentage points across all tested scales (2048-16384 samples) when features are pre-aligned by a vision-language pretraining objective. We provide a theoretical explanation grounded in sample complexity analysis: concatenation requires O(d_v + d_t) samples to learn its fusion projection, while cross-attention requires O(d_v * d_t) samples to learn bilinear attention weights, over 256 times as many for 512-dimensional CLIP features. When features are already aligned, the approximation error gap between the two methods vanishes, and concatenation's sample efficiency dominates at all practical dataset sizes. An alignment degradation study confirms a monotonic trend: as feature alignment degrades, concatenation's advantage grows from 1.3% to 2.8%. These findings provide a principled decision framework for fusion method selection in multimodal systems, with direct implications for the design of Multimodal Large Language Models.

多模态融合特征对齐拼接交叉注意力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。