通过净化再对齐,让单模态模型在缺失模态时仍保持鲁棒性。
Purify-then-Align: Towards Robust Human Sensing under Modality Missing with Knowledge Distillation from Noisy Multimodal Teacher
- 先用元学习动态过滤噪声模态,再用知识蒸馏对齐特征
- 在MM-Fi和XRF55数据集上显著提升单模态模型性能
- 适合处理多模态缺失场景的鲁棒人体感知任务
鲁棒的多模态人体感知面临模态缺失的关键挑战。主要障碍是异构数据间的表征差异以及低质量模态带来的污染效应。这两者存在因果关联:污染会从根本上阻碍表征差距的缩小。本文提出PTA框架,采用“净化-对齐”策略,通过元学习与知识扩散的协同机制解决这一因果依赖。首先,利用元学习驱动的加权机制动态降低噪声、低贡献模态的影响;随后,构建信息丰富的干净教师模型,通过基于扩散的知识蒸馏,优化各学生模态的特征表示。该策略最终生成具备跨模态知识的强单模态编码器。在大规模MM-Fi和XRF55数据集上,面对显著的表征差异与污染效应,PTA实现了最先进性能,显著提升了单模态模型在多种缺失模态场景下的鲁棒性。
原文摘要 · Abstract (English)
Robust multimodal human sensing must overcome the critical challenge of missing modalities. Two principal barriers are the Representation Gap between heterogeneous data and the Contamination Effect from low-quality modalities. These barriers are causally linked, as the corruption introduced by contamination fundamentally impedes the reduction of representation disparities. In this paper, we propose PTA, a novel "Purify-then-Align" framework that solves this causal dependency through a synergistic integration of meta-learning and knowledge diffusion. To purify the knowledge source, PTA first employs a meta-learning-driven weighting mechanism that dynamically learns to down-weight the influence of noisy, low-contributing modalities. Subsequently, to align different modalities, PTA introduces a diffusion-based knowledge distillation paradigm in which an information-rich clean teacher, formed from this purified consensus, refines the features of each student modality. The ultimate payoff of this "Purify-then-Align" strategy is the creation of exceptionally powerful single-modality encoders imbued with cross-modal knowledge. Comprehensive experiments on the large-scale MM-Fi and XRF55 datasets, under pronounced Representation Gap and Contamination Effect, demonstrate that PTA achieves state-of-the-art performance and significantly improves the robustness of single-modality models in diverse missing-modality scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。