用多个智能体模拟团队协作，提升视觉问答准确率
Multi-Agents Based on Large Language Models for Knowledge-based Visual Question Answering
- 设计三类角色智能体，按能力分配工具使用权限
- 在OK-VQA和A-OKVQA上分别提升2.2%和1.0%
- 适合需要多步推理与工具协同的复杂视觉问答任务
大型语言模型（LLMs）在基于知识的视觉问答（VQA）中取得了显著成果。然而现有方法仍存在两大挑战：无法自主使用外部工具，且缺乏团队协作能力。人类在面对问题时会根据熟悉程度决定是否使用工具——熟悉的问题可直接回答，陌生的问题则借助搜索引擎等工具。此外，人类也倾向于通过协作讨论获得更优答案。受此启发，我们提出多智能体投票框架。设计三类基于LLM的智能体，模拟团队中不同层级的成员，并根据其角色分配可用工具。各智能体分别给出答案，最终通过投票生成最终结果。在OK-VQA和A-OKVQA数据集上的实验表明，该方法分别优于其他基线2.2%和1.0%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved impressive results in knowledge-based Visual Question Answering (VQA). However existing methods still have challenges: the inability to use external tools autonomously, and the inability to work in teams. Humans tend to know whether they need to use external tools when they encounter a new question, e.g., they tend to be able to give a direct answer to a familiar question, whereas they tend to use tools such as search engines when they encounter an unfamiliar question. In addition, humans also tend to collaborate and discuss with others to get better answers. Inspired by this, we propose the multi-agent voting framework. We design three LLM-based agents that simulate different levels of staff in a team, and assign the available tools according to the levels. Each agent provides the corresponding answer, and finally all the answers provided by the agents are voted to get the final answer. Experiments on OK-VQA and A-OKVQA show that our approach outperforms other baselines by 2.2 and 1.0, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。