用自然语言查询交通视频，灵活高效找目标，速度比基线快9.6倍。
LAVA: Language Driven Scalable and Versatile Traffic Video Analytics
- 用自然语言直接提问，自动定位多粒度交通目标。
- 选择类查询F1提升14%，聚合类查询误差降低0.39，精度达86%。
- 适合城市交通管理、智能监控等需快速分析海量视频的场景。
在现代城市环境中，摄像头网络每天生成海量运行视频，总量可达拍字节级别，实现可扩展的视频分析至关重要。现有方法多采用基于SQL的查询范式，但受限于预定义语义类别，查询灵活性差。本文提出语言驱动的视频分析范式，构建系统Lava，支持自然语言输入，实现跨粒度、任意类别交通目标的检索。Lava包含三个核心组件：1）基于多臂赌博机的视频片段级高效采样方法；2）面向视频的开放世界检测模块，用于目标级检索；3）长时序目标轨迹提取方案，完整追踪目标运动路径。为全面评估，我们构建新基准，提供多样且语义丰富的自然语言谓词及细粒度标注。实验表明，Lava使选择类查询的F1分数提升14%，聚合类查询的MPAE降低0.39，顶k精度达86%，处理速度比最准确基线快9.6倍。代码与数据集已开源。
原文摘要 · Abstract (English)
In modern urban environments, camera networks generate massive amounts of operational footage -- reaching petabytes each day -- making scalable video analytics essential for efficient processing. Many existing approaches adopt an SQL-based paradigm for querying such large-scale video databases; however, this constrains queries to rigid patterns with predefined semantic categories, significantly limiting analytical flexibility. In this work, we explore a language-driven video analytics paradigm aimed at enabling flexible and efficient querying of high-volume video data driven by natural language. Particularly, we build \textsc{Lava}, a system that accepts natural language queries and retrieves traffic targets across multiple levels of granularity and arbitrary categories. \textsc{Lava} comprises three main components: 1) a multi-armed bandit-based efficient sampling method for video segment-level localization; 2) a video-specific open-world detection module for object-level retrieval; and 3) a long-term object trajectory extraction scheme for temporal object association, yielding complete trajectories for object-of-interests. To support comprehensive evaluation, we further develop a novel benchmark by providing diverse, semantically rich natural language predicates and fine-grained annotations for multiple videos. Experiments on this benchmark demonstrate that \textsc{Lava} improves $F_1$-scores for selection queries by $\mathbf{14\%}$, reduces MPAE for aggregation queries by $\mathbf{0.39}$, and achieves top-$k$ precision of $\mathbf{86\%}$, while processing videos $ \mathbf{9.6\times} $ faster than the most accurate baseline. Our code and dataset are available at https://github.com/yuyanrui/LAVA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。