提出轻量级融合层，解决多模态输入缺失时的鲁棒性与校准难题。
Robust Multimodal Learning via Entropy-Gated Contrastive Fusion
- 根据实例自适应调整熵系数，动态控制模态融合强度。
- 在50%缺失率下，掩码输入mAP提升18个百分点，ECE降低200%。
- 无需训练主干网络，可直接插入现有模型，适合工业部署。
现实中的多模态系统常面临输入缺失问题，例如机器人在工厂中丢失音频，或临床记录在推理时缺少检验数据。标准融合层难以同时兼顾鲁棒性与校准性。本文提出自适应熵门控对比融合（AECF），一种轻量级融合层，具备三个特性：(i) 每个样本自适应调节熵系数；(ii) 确保所有模态子集上的单调校准性；(iii) 直接从训练时的熵生成课程掩码。在AV-MNIST和MS-COCO数据集上，AECF在50%输入缺失率下，掩码输入mAP提升18个百分点，同时将ECE降低高达200%，仅增加1%运行时间。所有主干网络保持冻结，AECF可作为即插即用模块，实现鲁棒且校准良好的多模态推理。
原文摘要 · Abstract (English)
Real-world multimodal systems routinely face missing-input scenarios, and in reality, robots lose audio in a factory or a clinical record omits lab tests at inference time. Standard fusion layers either preserve robustness or calibration but never both. We introduce Adaptive Entropy-Gated Contrastive Fusion (AECF), a single light-weight layer that (i) adapts its entropy coefficient per instance, (ii) enforces monotone calibration across all modality subsets, and (iii) drives a curriculum mask directly from training-time entropy. On AV-MNIST and MS-COCO, AECF improves masked-input mAP by +18 pp at a 50% drop rate while reducing ECE by up to 200%, yet adds 1% run-time. All back-bones remain frozen, making AECF an easy drop-in layer for robust, calibrated multimodal inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。