无需训练即可整合多模态模型，实现低成本高效对话系统。
Training-Free Multimodal Large Language Model Orchestration
- 用大模型控制指令选择和调度各模态专家，实现可追溯的路由。
- 通过结构化记忆压缩多模态信息，减少重复调用专家次数。
- 支持全双工流式交互与中断响应，适合实际部署的多模态助手。
构建交互式全模态助手通常依赖端到端的多模态对齐，带来高昂的数据与计算成本，并限制可扩展性。本文提出无需训练的大语言模型编排（LLM Orchestration）框架，将现成的模态专家无缝集成到统一的多模态输入-输出系统中，无需额外梯度训练。该框架包含三个组件：(1) 大语言模型控制器，用于推断用户意图并生成显式的控制标记以选择和排序专家，实现协议约束且可审计的路由；(2) 以文本为中心的跨模态记忆，将多模态证据压缩为结构化记录，支持轻量级检索与复用，降低跨轮次冗余专家调用；(3) 统一交互层，执行路由与记忆决策，支持一致的模态转换、全双工流式通信及中断感知对话。在多个多模态基准测试中，该方法在标准评估条件下取得优异性能，同时保持低编排开销和模块化可升级性，为全模态系统提供低成本联合训练的实用替代方案。
原文摘要 · Abstract (English)
Building interactive omni-modal assistants often relies on end-to-end multimodal alignment to fuse heterogeneous modalities, which incurs substantial data and compute costs and limits extensibility. We present Training-Free Large Language Model Orchestration (LLM Orchestration), a training-free orchestration framework that integrates off-the-shelf modality experts into a unified multimodal input--output system without additional gradient-based training for integration. LLM Orchestration comprises three components: (1) an LLM controller that infers user intent and emits explicit control tokens for expert selection and sequencing, enabling protocol-constrained and auditable routing; (2) a text-centric cross-modal memory that compresses multimodal evidence into structured records for lightweight retrieval and reuse, reducing redundant expert invocations across turns; and (3) a unified interaction layer that executes routing and memory decisions to support consistent modality transitions, full-duplex streaming, and interruption-aware dialogue. Across diverse multimodal benchmarks, LLM Orchestration achieves strong performance under standard evaluation constraints while maintaining low orchestration overhead and modular upgradeability, providing a practical alternative to costly joint training for omni-modal systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。