解决多模态融合中噪声与缺失模态问题，提升模型鲁棒性。
DCER: Dual-Stage Compression and Energy-Based Reconstruction
- 双阶段压缩：模态内去噪+跨模态瓶颈强制真实融合
- 基于能量的重建使模型在模态缺失时仍能准确恢复表示
- 在三个数据集上表现领先，特别适用于高缺失场景
多模态融合面临两大鲁棒性挑战：噪声输入会降低表征质量，模态缺失会导致预测失败。我们提出DCER，一个统一框架，通过双阶段压缩和基于能量的重建来应对这两类问题。压缩阶段分两层：模态内频率变换（音频用小波，视频用DCT）去除噪声并保留任务相关模式；跨模态瓶颈令牌强制真实融合，避免仅依赖特定模态的捷径。对于缺失模态，基于能量的重建通过在学习到的能量函数上进行梯度下降恢复表征，最终能量值提供内在不确定性量化（与预测误差相关性ρ > 0.72）。在CMU-MOSI、CMU-MOSEI和CH-SIMS上的实验表明，该方法在所有基准上均达到最先进性能，展现出在完整输入和高缺失条件下均优异的鲁棒性，呈U型趋势。代码将开源于Github。
原文摘要 · Abstract (English)
Multimodal fusion faces two robustness challenges: noisy inputs degrade representation quality, and missing modalities cause prediction failures. We propose DCER, a unified framework addressing both challenges through dual-stage compression and energy-based reconstruction. The compression stage operates at two levels: within-modality frequency transforms (wavelet for audio, DCT for video) remove noise while preserving task-relevant patterns, and cross-modality bottleneck tokens force genuine integration rather than modality-specific shortcuts. For missing modalities, energy-based reconstruction recovers representations via gradient descent on a learned energy function, with the final energy providing intrinsic uncertainty quantification (\r{ho} > 0.72 correlation with prediction error). Experiments on CMU-MOSI, CMU-MOSEI, and CH-SIMS demonstrate state-of-the-art performance across all benchmarks, with a U-shaped robustness pattern favoring multimodal fusion at both complete and high-missing conditions. The code will be available on Github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。