提出跨层次语义协同表示,解决多模态融合中的语义错位问题。
CLCR: Cross-Level Semantic Collaborative Representation for Multimodal Learning
- 构建三层次语义结构,分层约束跨模态交互。
- 在各层级共享空间内交换语义,避免私有信息泄露。
- 适用于情感识别、动作识别等多任务多模态场景。
多模态学习旨在捕捉不同模态间的共享与私有信息。然而,现有方法将所有模态投影到单一潜在空间进行融合,常忽略多模态数据异步且多层次的语义结构,导致语义错位与误差传播,降低表示质量。为此,我们提出跨层次协同表示(CLCR),显式地将每种模态特征组织为三个语义层级,并为跨模态交互设定层级约束。首先,语义层级编码器对浅层、中层和深层特征进行跨模态对齐,建立交互基础。随后,在每个层级上,内部协同交换域(IntraCED)将特征分解为共享与私有子空间,并通过可学习的令牌预算限制跨模态注意力仅作用于共享子空间,确保仅共享语义被交换,防止私有通道泄漏。为整合跨层级信息,层级间协同聚合域(InterCAD)使用可学习锚点同步语义尺度,选择性融合共享表示并门控私有线索,生成紧凑的任务表示。我们进一步引入正则化项以强制共享与私有特征分离,并最小化跨层级干扰。在涵盖情绪识别、事件定位、情感分析和动作识别的六个基准上实验表明,CLCR性能优异且具备良好泛化能力。
原文摘要 · Abstract (English)
Multimodal learning aims to capture both shared and private information from multiple modalities. However, existing methods that project all modalities into a single latent space for fusion often overlook the asynchronous, multi-level semantic structure of multimodal data. This oversight induces semantic misalignment and error propagation, thereby degrading representation quality. To address this issue, we propose Cross-Level Co-Representation (CLCR), which explicitly organizes each modality's features into a three-level semantic hierarchy and specifies level-wise constraints for cross-modal interactions. First, a semantic hierarchy encoder aligns shallow, mid, and deep features across modalities, establishing a common basis for interaction. And then, at each level, an Intra-Level Co-Exchange Domain (IntraCED) factorizes features into shared and private subspaces and restricts cross-modal attention to the shared subspace via a learnable token budget. This design ensures that only shared semantics are exchanged and prevents leakage from private channels. To integrate information across levels, the Inter-Level Co-Aggregation Domain (InterCAD) synchronizes semantic scales using learned anchors, selectively fuses the shared representations, and gates private cues to form a compact task representation. We further introduce regularization terms to enforce separation of shared and private features and to minimize cross-level interference. Experiments on six benchmarks spanning emotion recognition, event localization, sentiment analysis, and action recognition show that CLCR achieves strong performance and generalizes well across tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。