arXiv:2410.01506cs.CVcs.AI2024-10ICLR被引 33

用可学习的图操作融合多模态特征,提升跨域融合效果

Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion

  • 构建多粒度关系图,将特征映射到可解释的图空间
  • 通过迭代更新图关系,实现深层特征交互融合
  • 适合需要跨模态、跨域特征融合的视觉任务

在计算机视觉任务中,特征常来自不同表示、领域(如室内与室外)和模态(如文本、图像、视频)。有效融合这些特征对性能至关重要,尤其在视觉-语言模型等预训练模型可用的背景下。然而,常见的融合方法(如拼接、逐元素操作、非线性变换)往往难以捕捉结构关系、深层特征交互,且存在效率低或跨域/跨模态特征对齐问题。本文提出从高维特征空间转向低维可解释的图空间,构建编码不同层次(如clip、frame、patch、token等)特征关系的关系图。为捕获更深层次交互,通过迭代图关系更新扩展图结构,并引入可学习的图融合算子整合这些扩展关系,实现更有效的融合。该方法以关系为中心,在同质空间中操作,数学上具有原理性,类似于通过多重线性多项式进行逐元素关系评分聚合。我们在视频异常检测任务上验证了该图融合方法的有效性,展示了在多表示、多模态、多领域特征融合任务中的优异表现。

原文摘要 · Abstract (English)

In computer vision tasks, features often come from diverse representations, domains (e.g., indoor and outdoor), and modalities (e.g., text, images, and videos). Effectively fusing these features is essential for robust performance, especially with the availability of powerful pre-trained models like vision-language models. However, common fusion methods, such as concatenation, element-wise operations, and non-linear techniques, often fail to capture structural relationships, deep feature interactions, and suffer from inefficiency or misalignment of features across domains or modalities. In this paper, we shift from high-dimensional feature space to a lower-dimensional, interpretable graph space by constructing relationship graphs that encode feature relationships at different levels, e.g., clip, frame, patch, token, etc. To capture deeper interactions, we expand graphs through iterative graph relationship updates and introduce a learnable graph fusion operator to integrate these expanded relationships for more effective fusion. Our approach is relationship-centric, operates in a homogeneous space, and is mathematically principled, resembling element-wise relationship score aggregation via multilinear polynomials. We demonstrate the effectiveness of our graph-based fusion method on video anomaly detection, showing strong performance across multi-representational, multi-modal, and multi-domain feature fusion tasks.

图神经网络多模态融合特征融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。