解决多模态模型偏科问题，让不同模态公平协作
RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer
- 用轮转查询机制打破注意力偏好固化循环
- 在多个数据集上显著提升多模态融合效果
- 适合研究多模态融合与模型公平性的学者
多模态学习在融合不同质量的模态信息时面临挑战。尽管自注意力机制能动态调整各模态权重，但实验发现其自适应能力会逐渐减弱，模型趋向于固定偏好某一模态，形成自我强化循环，导致注意力键值分布差距扩大，抑制了动态融合能力。为此，本文提出简单的滚动查询（RollingQ）方法，通过轮转查询来打破该循环，平衡模态间注意力分配。在多种多模态场景下的大量实验验证了该方法的有效性，恢复合作动态对提升主流多模态Transformer的综合性能至关重要。代码已开源。
原文摘要 · Abstract (English)
Multimodal learning faces challenges in effectively fusing information from diverse modalities, especially when modality quality varies across samples. Dynamic fusion strategies, such as attention mechanism in Transformers, aim to address such challenge by adaptively emphasizing modalities based on the characteristics of input data. However, through amounts of carefully designed experiments, we surprisingly observed that the dynamic adaptability of widely-used self-attention models diminishes. Model tends to prefer one modality regardless of data characteristics. This bias triggers a self-reinforcing cycle that progressively overemphasizes the favored modality, widening the distribution gap in attention keys across modalities and deactivating attention mechanism's dynamic properties. To revive adaptability, we propose a simple yet effective method Rolling Query (RollingQ), which balances attention allocation by rotating the query to break the self-reinforcing cycle and mitigate the key distribution gap. Extensive experiments on various multimodal scenarios validate the effectiveness of RollingQ and the restoration of cooperation dynamics is pivotal for enhancing the broader capabilities of widely deployed multimodal Transformers. The source code is available at https://github.com/GeWu-Lab/RollingQ_ICML2025.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。