智能路由系统动态选模，高效精准检索视频内容。
Smart Routing for Multimodal Video Retrieval: When to Search What
- 基于大模型分析查询意图，自动选择最优检索模态。
- 计算量减少41%，召回率仍达60.9% Recall@5。
- 适合需低成本部署的实时视频检索系统使用。
我们提出 ModaRoute，一个基于大语言模型的智能路由系统，可动态选择多模态视频检索中的最优模态。尽管密集文本字幕可实现75.9% Recall@5，但需昂贵离线处理，且在34%含场景文字的视频中遗漏关键视觉信息。ModaRoute通过分析查询意图与信息需求，将每条查询平均调用1.78个模态，远低于全量3.0模态搜索，计算开销降低41%，同时保持60.9% Recall@5。系统在180万视频片段上验证，证明智能路由能有效降低基础设施成本，为实际部署提供可行方案。
原文摘要 · Abstract (English)
We introduce ModaRoute, an LLM-based intelligent routing system that dynamically selects optimal modalities for multimodal video retrieval. While dense text captions can achieve 75.9% Recall@5, they require expensive offline processing and miss critical visual information present in 34% of clips with scene text not captured by ASR. By analyzing query intent and predicting information needs, ModaRoute reduces computational overhead by 41% while achieving 60.9% Recall@5. Our approach uses GPT-4.1 to route queries across ASR (speech), OCR (text), and visual indices, averaging 1.78 modalities per query versus exhaustive 3.0 modality search. Evaluation on 1.8M video clips demonstrates that intelligent routing provides a practical solution for scaling multimodal retrieval systems, reducing infrastructure costs while maintaining competitive effectiveness for real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。