用动态专家聚合实现跨模态推理,无需训练即可适配多种任务。
MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation
- 根据输入模态和任务需求动态选择专业专家模型
- 在多个基准上优于强基线,医学、视频等任务均提升显著
- 适合需要灵活、透明多模态推理的场景,如医疗与金融分析
融合预训练专家模型为可扩展的多模态推理提供了巨大潜力,但统一框架的构建仍面临挑战,尤其在输入模态多样性和任务复杂性增加的情况下。例如,医学诊断需对结构化临床表格进行精确推理,而金融预测依赖于图表数据的解读以做出判断。为此,我们提出MEXA——一种无需训练的框架,通过模态与任务感知的专家模型动态聚合,实现跨多样且异构领域的有效多模态推理。MEXA根据输入模态和任务特定推理需求(即技能)动态选择专家模型。每个专家模型专精于特定模态-任务组合,生成可解释的文本推理输出。随后,使用大推理模型(LRM)对这些输出进行聚合与进一步推理,生成最终答案。该模块化设计可在无额外训练开销的前提下,实现跨领域灵活透明的多模态推理。我们在包括视频推理、音频推理、3D理解及医学问答在内的多个多模态基准上进行了广泛评估,MEXA在各项任务中持续优于强基线,验证了其专家驱动选择与聚合策略的有效性与普适性。
原文摘要 · Abstract (English)
Combining pre-trained expert models offers substantial potential for scalable multimodal reasoning, but building a unified framework remains challenging due to the increasing diversity of input modalities and task complexity. For instance, medical diagnosis requires precise reasoning over structured clinical tables, while financial forecasting depends on interpreting plot-based data to make informed predictions. To tackle this challenge, we introduce MEXA, a training-free framework that performs modality- and task-aware aggregation of multiple expert models to enable effective multimodal reasoning across diverse and distinct domains. MEXA dynamically selects expert models based on the input modality and the task-specific reasoning demands (i.e., skills). Each expert model, specialized in a modality task pair, generates interpretable textual reasoning outputs. MEXA then aggregates and reasons over these outputs using a Large Reasoning Model (LRM) to produce the final answer. This modular design allows flexible and transparent multimodal reasoning across diverse domains without additional training overhead. We extensively evaluate our approach on diverse multimodal benchmarks, including Video Reasoning, Audio Reasoning, 3D Understanding, and Medical QA. MEXA consistently delivers performance improvements over strong multimodal baselines, highlighting the effectiveness and broad applicability of our expert-driven selection and aggregation in diverse multimodal reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。