让视频大模型持续学习新工具，不遗忘旧知识。
COLT: Enhancing Video Large Language Models with Continual Tool Usage
- 用可学习的工具代码本作为记忆，动态匹配指令与工具特征。
- 在流式工具数据下仍保持对旧工具的记忆，避免灾难性遗忘。
- 适用于需要持续学习新技能的智能视频分析场景。
大型语言模型（LLMs）的成功显著推动了视频理解的研究。为充分利用训练良好的专家模型（即工具），视频大模型聚焦于工具使用能力的探索。现有方法要么提示闭源LLM，要么采用指令微调范式进行工具使用微调。然而，这些方法假设工具库是固定的，难以泛化到工具数据持续演化和流式输入的真实环境。为此，我们提出开放源代码的视频大模型持续工具使用增强方法（COLT），可在连续工具流中自动获取工具使用能力，且不会导致过去所学工具的“灾难性遗忘”。具体而言，COLT引入可学习的工具代码本作为特定工具的记忆系统，并根据用户指令与代码本中工具特征的相似度动态选择相关工具。为释放视频大模型的工具使用潜力，我们构建了一个以视频为中心的工具使用指令微调数据集VideoToolBench。在多个既有视频大模型基准和专用于工具使用的VideoToolBench数据集上的大量实验表明，所提COLT方法达到当前最优性能。
原文摘要 · Abstract (English)
The success of Large Language Models (LLMs) has significantly propelled the research of video understanding. To harvest the benefits of well-trained expert models (i.e., tools), video LLMs prioritize the exploration of tool usage capabilities. Existing methods either prompt closed-source LLMs or employ the instruction tuning paradigm for tool-use fine-tuning. These methods, however, assume an established repository of fixed tools and struggle to generalize to real-world environments where tool data is perpetually evolving and streaming in. To this end, we propose to enhance open-source video LLMs with COntinuaL Tool usage (termed COLT), which automatically acquires tool-use ability in a successive tool stream without suffering 'catastrophic forgetting' of the past learned tools. Specifically, our COLT incorporates a learnable tool codebook as a tool-specific memory system. Then relevant tools are dynamically selected based on the similarity between user instruction and tool features within the codebook. To unleash the tool usage potential of video LLMs, we collect a video-centric tool-use instruction tuning dataset VideoToolBench. Extensive experiments on both previous video LLM benchmarks and the tool-use-specific VideoToolBench dataset demonstrate the state-of-the-art performance of our proposed COLT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。