arXiv:2601.06453cs.AI2026-01ACL被引 5

用多个专家代理协作解析多模态传感器数据,提升准确率与鲁棒性。

ConSensus: Multi-Agent Collaboration for Multimodal Sensing

  • 将多模态任务拆分为专注特定模态的代理,分头处理
  • 融合语义与统计共识,使结果在缺损数据下仍可靠,平均提效7.1%
  • 无需迭代辩论,单轮融合降低12.7倍计算成本,适合实际部署

大型语言模型(LLMs)正越来越多地与传感器数据结合,以感知人体生理状态和物理世界。然而,准确解读异构多模态传感器数据仍是核心挑战。我们发现,单一的大型语言模型常在跨模态推理中表现不连贯,导致理解不全并受先验知识干扰。为此,我们提出ConSensus——一种无需训练的多代理协作框架,将多模态感知任务分解为专门化、模态感知的代理。为聚合代理输出,我们设计了一种混合融合机制,兼顾语义聚合(支持跨模态推理与上下文理解)与统计共识(通过模态间一致性增强鲁棒性)。二者互补失效模式,可在传感器噪声或数据缺失条件下实现可靠推断。我们在五个多样化的多模态感知基准上评估,结果显示,相比单代理基线,平均准确率提升7.1%。此外,ConSensus在性能上达到或超过迭代式多代理辩论方法,同时通过单轮混合融合协议,平均融合令牌开销降低12.7倍,提供高效且稳健的真实世界多模态感知解决方案。源代码见:https://github.com/nokia/multi-agent-collaboration-for-multimodal-sensing。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly grounded in sensor data to perceive and reason about human physiology and the physical world. However, accurately interpreting heterogeneous multimodal sensor data remains a fundamental challenge. We show that a single monolithic LLM often fails to reason coherently across modalities, leading to incomplete interpretations and prior-knowledge bias. We introduce ConSensus, a training-free multi-agent collaboration framework that decomposes multimodal sensing tasks into specialized, modality-aware agents. To aggregate agent-level interpretations, we propose a hybrid fusion mechanism that balances semantic aggregation, which enables cross-modal reasoning and contextual understanding, with statistical consensus, which provides robustness through agreement across modalities. While each approach has complementary failure modes, their combination enables reliable inference under sensor noise and missing data. We evaluate ConSensus on five diverse multimodal sensing benchmarks, demonstrating an average accuracy improvement of 7.1% over the single-agent baseline. Furthermore, ConSensus matches or exceeds the performance of iterative multi-agent debate methods while achieving a 12.7 times reduction in average fusion token cost through a single-round hybrid fusion protocol, yielding a robust and efficient solution for real-world multimodal sensing tasks. The source code is available at https://github.com/nokia/multi-agent-collaboration-for-multimodal-sensing.

多模态智能传感多智能体融合算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。