通过提升融合表示的秩,同时解决多模态特征与模态崩溃问题。
Countering Multi-modal Representation Collapse through Rank-targeted Fusion
- 基于有效秩设计可选融合机制,增强跨模态信息互补性。
- 在NTURGBD等数据集上性能提升最高达3.74%,优于现有方法。
- 适合需要融合深度与RGB视频的动作预测任务。
多模态融合常面临两种表示崩溃:特征崩溃(维度丧失判别力,由特征谱衡量)与模态崩溃(某一模态主导其余)。依赖多传感器数据的人体动作预测受此双重制约。现有方法分别应对两类崩溃,缺乏统一框架。本文提出有效秩作为量化并缓解两类崩溃的统一指标,设计理论支撑的融合框架「Rank-enhancing Token Fuser」,选择性融合低信息量特征与互补特征。实验表明该方法显著提升融合表示的有效秩。针对模态崩溃,发现深度图与RGB融合能维持表征平衡。在动作预测任务中,提出基于深度感知的融合框架「R3D」,在NTURGBD、UTKinect和DARai数据集上性能超越现有最优方法,最高提升3.74%。代码已开源。
原文摘要 · Abstract (English)
Multi-modal fusion methods often suffer from two types of representation collapse: feature collapse where individual dimensions lose their discriminative power (as measured by eigenspectra), and modality collapse where one dominant modality overwhelms the other. Applications like human action anticipation that require fusing multifarious sensor data are hindered by both feature and modality collapse. However, existing methods attempt to counter feature collapse and modality collapse separately. This is because there is no unifying framework that efficiently addresses feature and modality collapse in conjunction. In this paper, we posit the utility of effective rank as an informative measure that can be utilized to quantify and counter both the representation collapses. We propose \textit{Rank-enhancing Token Fuser}, a theoretically grounded fusion framework that selectively blends less informative features from one modality with complementary features from another modality. We show that our method increases the effective rank of the fused representation. To address modality collapse, we evaluate modality combinations that mutually increase each others' effective rank. We show that depth maintains representational balance when fused with RGB, avoiding modality collapse. We validate our method on action anticipation, where we present \texttt{R3D}, a depth-informed fusion framework. Extensive experiments on NTURGBD, UTKinect, and DARai demonstrate that our approach significantly outperforms prior state-of-the-art methods by up to 3.74\%. Our code is available at: \href{https://github.com/olivesgatech/R3D}{https://github.com/olivesgatech/R3D}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。