用可训练的分层自动机系统,让多个智能体协作解视觉推理题。
MATA: A Trainable Hierarchical Automaton System for Multi-Agent Visual Reasoning
- 分层自动机架构,顶层由可训练超智能体决定任务切换
- 在多个基准上超越单体与组合模型,准确率领先10%以上
- 适合需要透明推理过程的高可靠性视觉任务场景
近期视觉语言模型虽具备强感知能力,但其隐式推理难以解释且易产生幻觉。组合方法提升可解释性,但多数依赖单一智能体或人工设计流程,无法判断何时应协作或竞争。我们提出MATA(多智能体分层可训练自动机),一种以分层有限状态自动机形式呈现的多智能体系统,其顶层状态转移由可训练的超智能体决定。每个智能体对应超自动机中的一个状态,并运行小型基于规则的子自动机实现可靠微观控制。所有智能体共享内存,生成透明的执行历史。为监督超智能体的转移策略,我们构建转移轨迹树,转换为记忆到下一状态对,形成MATA-SFT-90K数据集用于监督微调。微调后的LLM作为转移策略,能理解查询和智能体能力,高效选择最优智能体完成任务。在多个视觉推理基准上,MATA性能优于单体与组合基线,达到当前最佳水平。代码与数据集见https://github.com/ControlNet/MATA。
原文摘要 · Abstract (English)
Recent vision-language models have strong perceptual ability but their implicit reasoning is hard to explain and easily generates hallucinations on complex queries. Compositional methods improve interpretability, but most rely on a single agent or hand-crafted pipeline and cannot decide when to collaborate across complementary agents or compete among overlapping ones. We introduce MATA (Multi-Agent hierarchical Trainable Automaton), a multi-agent system presented as a hierarchical finite-state automaton for visual reasoning whose top-level transitions are chosen by a trainable hyper agent. Each agent corresponds to a state in the hyper automaton, and runs a small rule-based sub-automaton for reliable micro-control. All agents read and write a shared memory, yielding transparent execution history. To supervise the hyper agent's transition policy, we build transition-trajectory trees and transform to memory-to-next-state pairs, forming the MATA-SFT-90K dataset for supervised finetuning (SFT). The finetuned LLM as the transition policy understands the query and the capacity of agents, and it can efficiently choose the optimal agent to solve the task. Across multiple visual reasoning benchmarks, MATA achieves the state-of-the-art results compared with monolithic and compositional baselines. The code and dataset are available at https://github.com/ControlNet/MATA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。