提出自洽梯度场融合模型,解决多模态信息混合导致的训练不稳问题。
Multimodal Fusion via Self-Consistent Task-Gradient Fields
- 用自洽场原理分离任务梯度与特征重构,避免优化冲突
- 在音频-视觉-文本等任务中表现鲁棒,支持缺失数据和不同输入大小
- 结构简单却有效,适合需要稳定多模态融合的场景
多模态学习旨在从不同输入中尽可能保留与任务相关的信息。然而,现有融合设计常扭曲反馈回路至特征提取器,激进地合并模态会纠缠其表征,使特征提取器对不完整输入敏感。同时,通过辅助损失尝试分离特征,又常引入优化冲突,分散主任务注意力。本文提出自洽场自编码器(SCFAE),为任务梯度提供更优路径。方法遵循自洽场原则,在任务学习与特征组织间平衡,从而最小化互信息。每个模态使用小型自编码器以保持信息完整。任务损失作为驱动力选择预测性特征,重构损失则作为约束将特征分离至独立子空间。双重目标通过互补特征子空间协同运作,缓解优化干扰。我们在音频-视觉-文本、音频-视觉及图像-视频基准上评估SCFAE,结果表明其通过简洁结构对缺失数据和不均衡输入更具鲁棒性。梯度分析证实,SCFAE避免了优化冲突,维持了稳定的训练动态。
原文摘要 · Abstract (English)
Multimodal learning aims to preserve as much task-related information as possible from different inputs. However, current fusion designs often distort the feedback loop to feature extractors. Aggressively merging modalities entangles their representations, making the feature extractors fragile to incomplete inputs. Meanwhile, attempting to separate features via auxiliary losses frequently introduces optimization conflicts that distract from the primary task. We propose the Self-Consistent Field Autoencoder (SCFAE) to provide a better path for task gradients. Our method follows the self-consistent field principle to balance task learning with feature organization, thereby minimizing mutual information. We use small autoencoders for each modality to keep information intact. The task loss acts as a driving force to select predictive features. The reconstruction loss acts as a constraint to separate these features into independent subspaces. These dual objectives operate through complementary feature subspaces, thereby mitigating optimization interference. We evaluate SCFAE on audio-visual-text, audio-visual, and image-video benchmarks. Results show that SCFAE handles missing data and unequal input sizes more robustly via a simple structure. Gradient analysis confirms that SCFAE avoids conflicts and maintains stable training dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。