arXiv:2507.02985cs.CVcs.AI2025-07被引 5

提出一种线性可扩展的多模态融合方法,解决高模态场景下的计算瓶颈。

Gated Recursive Fusion: A Stateful Approach to Scalable Multimodal Transformers

  • 采用递归管道逐个处理模态,用门控单元动态控制信息流动。
  • 在CMU-MOSI上表现接近复杂基线模型,且计算复杂度为O(n)。
  • 适合高模态、需高效融合的场景,如多传感器或跨模态理解任务。

多模态学习面临深层细粒度融合与计算可扩展性之间的根本矛盾。尽管交叉注意力模型通过全面的成对融合取得优异性能,但其二次复杂度在多模态场景下难以承受。本文提出门控递归融合(GRF),一种新型架构,在线性可扩展的递归流程中实现交叉模态注意力的强大能力。该方法按顺序处理各模态,每步更新一个动态演化的多模态上下文向量。核心是基于Transformer解码器层构建的融合模块,执行对称交叉注意力,相互丰富共享上下文与输入模态。随后通过门控融合单元(GFU)——受GRU启发的机制——动态调节信息流,实现特征的选择性保留或丢弃。这种具有状态的递归设计使复杂度线性增长,达到O(n),适用于高模态环境。在CMU-MOSI基准上的实验表明,GRF性能优于或接近更复杂的基线模型。嵌入空间可视化显示,其渐进式融合机制生成结构清晰、类别可分的表示。本工作为强大而高效的多模态表征学习提供了一种稳健范式。

原文摘要 · Abstract (English)

Multimodal learning faces a fundamental tension between deep, fine-grained fusion and computational scalability. While cross-attention models achieve strong performance through exhaustive pairwise fusion, their quadratic complexity is prohibitive for settings with many modalities. We address this challenge with Gated Recurrent Fusion (GRF), a novel architecture that captures the power of cross-modal attention within a linearly scalable, recurrent pipeline. Our method processes modalities sequentially, updating an evolving multimodal context vector at each step. The core of our approach is a fusion block built on Transformer Decoder layers that performs symmetric cross-attention, mutually enriching the shared context and the incoming modality. This enriched information is then integrated via a Gated Fusion Unit (GFU) a GRU-inspired mechanism that dynamically arbitrates information flow, enabling the model to selectively retain or discard features. This stateful, recurrent design scales linearly with the number of modalities, O(n), making it ideal for high-modality environments. Experiments on the CMU-MOSI benchmark demonstrate that GRF achieves competitive performance compared to more complex baselines. Visualizations of the embedding space further illustrate that GRF creates structured, class-separable representations through its progressive fusion mechanism. Our work presents a robust and efficient paradigm for powerful, scalable multimodal representation learning.

多模态递归融合线性复杂度Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。