用视觉语言模型实现视频多模态搜索，支持对话式交互。
V-Agent: An Interactive Video Search System Using Vision-Language Models
- 三智能体协作：路由、搜索、对话，提升理解与响应能力。
- 在MultiVENT 2.0上达到顶尖零样本性能，优于传统文本检索。
- 适合需要跨模态视频检索的研究者与开发者使用。
我们提出V-Agent，一个基于视觉语言模型（VLM）的多智能体视频搜索系统，支持交互式对话。通过微调小规模视频偏好数据集，并融合图像-文本检索模型的向量表示，该系统克服了传统文本检索在多模态场景下的局限性。VLM将视频帧和语音识别（ASR）生成的字幕嵌入共享多模态空间，实现对视觉与听觉内容的联合理解。系统包含路由、搜索与聊天三个智能体，协同处理用户意图，优化搜索结果并进行自然对话。搜索模块结合VLM检索模型与重排序模块，进一步提升检索质量。在MultiVENT 2.0基准上，该框架实现最先进的零样本性能，展示了其在学术研究与实际应用中的潜力。检索模型与演示视频已开源至https://huggingface.co/NCSOFT/multimodal-embedding。
原文摘要 · Abstract (English)
We introduce V-Agent, a novel multi-agent platform designed for advanced video search and interactive user-system conversations. By fine-tuning a vision-language model (VLM) with a small video preference dataset and enhancing it with a retrieval vector from an image-text retrieval model, we overcome the limitations of traditional text-based retrieval systems in multimodal scenarios. The VLM-based retrieval model independently embeds video frames and audio transcriptions from an automatic speech recognition (ASR) module into a shared multimodal representation space, enabling V-Agent to interpret both visual and spoken content for context-aware video search. This system consists of three agents-a routing agent, a search agent, and a chat agent-that work collaboratively to address user intents by refining search outputs and communicating with users. The search agent utilizes the VLM-based retrieval model together with an additional re-ranking module to further enhance video retrieval quality. Our proposed framework demonstrates state-of-the-art zero-shot performance on the MultiVENT 2.0 benchmark, highlighting its potential for both academic research and real-world applications. The retrieval model and demo videos are available at https://huggingface.co/NCSOFT/multimodal-embedding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。