用专家混合模型补全缺失模态,提升动作质量评估鲁棒性
MCMoE: Completing Missing Modalities with Mixture of Experts for Incomplete Multimodal Action Quality Assessment
- 设计自适应门控生成器动态重建缺失模态
- 通过专家混合机制融合单模态知识并生成联合表征
- 在三个公开数据集上实现完整与不完整场景下的最佳性能
多模态动作质量评估(AQA)近年来成为有前景的方向,通过共享上下文线索的互补信息,增强对高度相似动作序列中细微类内差异的判别能力。然而,在实际推理阶段,部分模态常缺失,导致现有模型无法运行,并因跨模态交互中断引发性能灾难性下降。为此,我们提出一种统一单阶段训练的缺失模态补全框架MCMoE。设计自适应门控模态生成器,动态融合可用信息以重建缺失模态;引入模态专家学习单模态知识,并动态混合所有专家知识提取跨模态联合表征。利用专家混合机制进一步精炼和补充缺失模态。训练阶段,挖掘完整多模态特征与单模态专家知识,指导模态生成与基于生成的联合表示提取。大量实验表明,MCMoE在三个公开AQA基准上,无论是完整还是不完整多模态学习场景均达到领先效果。代码已开源。
原文摘要 · Abstract (English)
Multimodal Action Quality Assessment (AQA) has recently emerged as a promising paradigm. By leveraging complementary information across shared contextual cues, it enhances the discriminative evaluation of subtle intra-class variations in highly similar action sequences. However, partial modalities are frequently unavailable at the inference stage in reality. The absence of any modality often renders existing multimodal models inoperable. Furthermore, it triggers catastrophic performance degradation due to interruptions in cross-modal interactions. To address this issue, we propose a novel Missing Completion Framework with Mixture of Experts (MCMoE) that unifies unimodal and joint representation learning in single-stage training. Specifically, we propose an adaptive gated modality generator that dynamically fuses available information to reconstruct missing modalities. We then design modality experts to learn unimodal knowledge and dynamically mix the knowledge of all experts to extract cross-modal joint representations. With a mixture of experts, missing modalities are further refined and complemented. Finally, in the training phase, we mine the complete multimodal features and unimodal expert knowledge to guide modality generation and generation-based joint representation extraction. Extensive experiments demonstrate that our MCMoE achieves state-of-the-art results in both complete and incomplete multimodal learning on three public AQA benchmarks. Code is available at https://github.com/XuHuangbiao/MCMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。