让多模态模型学会抵抗模态失效，提升真实场景可靠性
ModalImmune: Immunity Driven Unlearning via Self Destructive Training
- 训练时主动破坏特定模态数据，迫使模型学得鲁棒联合表征
- 在多个基准上验证，模型对模态丢失和损坏的抗性显著提升
- 适合需要高可靠性的多模态应用，如医疗、自动驾驶
多模态系统在部署时可能遭遇部分或完全的输入模态丢失，影响实际应用中的可靠性。本文提出ModalImmune训练框架，通过有意识且可控地在训练中坍塌选定模态信息，使模型学习到对破坏性模态影响具有鲁棒性的联合表征。该框架结合谱自适应坍塌正则化、基于信息增益的控制器实现精准干预、曲率感知梯度掩码以稳定破坏性更新，以及经认证的Neumann截断超梯度方法实现元参数自动调节。在标准多模态基准上的实证评估表明，ModalImmune在保持收敛稳定性与重建能力的同时，显著提升了模型对模态移除和污染的韧性。
原文摘要 · Abstract (English)
Multimodal systems are vulnerable to partial or complete loss of input channels at deployment, which undermines reliability in real-world settings. This paper presents ModalImmune, a training framework that enforces modality immunity by intentionally and controllably collapsing selected modality information during training so the model learns joint representations that are robust to destructive modality influence. The framework combines a spectrum-adaptive collapse regularizer, an information-gain guided controller for targeted interventions, curvature-aware gradient masking to stabilize destructive updates, and a certified Neumann-truncated hyper-gradient procedure for automatic meta-parameter adaptation. Empirical evaluation on standard multimodal benchmarks demonstrates that ModalImmune improves resilience to modality removal and corruption while retaining convergence stability and reconstruction capacity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。