通过多轮动态协作的AI代理团队,显著提升长视频理解准确率。
LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agents
- 构建多代理协作框架,按任务动态组队并迭代讨论。
- 在四个主流任务上达80%准确率,LongVideoBench提升13.3%。
- 首个超越闭源与开源模型的长视频理解系统,适合复杂视频分析场景。
现有多模态大模型在建模长视频时序上下文方面面临挑战。当前主流代理方法依赖外部工具辅助单个模型回答长视频问题,但单一模型仍难以全面理解,导致性能受限。为此,我们提出首个支持多轮动态协作的多模态大模型代理框架LVAgent。该方法包含四步:1)选择:根据任务从模型库中预选最优代理组合;2)感知:设计高效检索策略,覆盖关键时序片段,兼顾计算效率;3)行动:代理回答问题并交换推理依据;4)反思:评估每轮表现,动态优化代理团队。通过多轮协作,代理持续改进答案。LVAgent在四个主流长视频理解任务上达到80%准确率,尤其在LongVideoBench上相较之前方法提升13.3%,超越所有闭源(如GPT-4o)和开源模型(如InternVL-2.5、Qwen2-VL)。代码已开源:https://github.com/64327069/LVAgent。
原文摘要 · Abstract (English)
Existing MLLMs encounter significant challenges in modeling the temporal context within long videos. Currently, mainstream Agent-based methods use external tools to assist a single MLLM in answering long video questions. Despite such tool-based support, a solitary MLLM still offers only a partial understanding of long videos, resulting in limited performance. In order to better address long video tasks, we introduce LVAgent, the first framework enabling multi-round dynamic collaboration of MLLM agents in long video understanding. Our method consists of four key steps: 1) Selection: We pre-select appropriate agents from the model library to form optimal agent teams based on different tasks. 2) Perception: We design an effective retrieval scheme for long videos to improve the coverage of critical temporal segments while maintaining computational efficiency. 3) Action: Agents answer long video questions and exchange reasons. 4) Reflection: We evaluate each agent's performance in each round of discussion and optimize the agent team for dynamic collaboration. The agents iteratively refine their answers by multi-round dynamical collaboration of MLLM agents. LVAgent is the first agent system method that outperforms all closed-source models (like GPT-4o) and open-source models (like InternVL-2.5 and Qwen2-VL) in the long video understanding tasks. Our LVAgent achieves an accuracy of 80\% on four mainstream long video understanding tasks. Notably, LVAgent improves accuracy by 13.3\% on LongVideoBench. Code is available at https://github.com/64327069/LVAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。