arXiv:2512.07344cs.DCcs.AI2025-12中稿 · IEEE International…被引 4

将视频理解的内存与检索下沉到边缘,实现秒级响应。

Venus: An Efficient Edge Memory-and-Retrieval System for VLM-based Online Video Understanding

  • 边缘端构建分层记忆,自动提取关键帧并嵌入
  • 查询阶段采用阈值渐进采样,提升多样性与效率
  • 相比顶尖方法提速15至131倍,适合实时视频应用

视觉语言模型(VLMs)在多模态理解中表现出色,正被广泛应用于在线视频理解。然而,现有研究多关注模型推理能力,忽视部署开销,导致实际应用中系统负担过重。为此,我们提出Venus,一种面向高效在线视频理解的边缘端记忆与检索系统。Venus采用边缘-云解耦架构,将记忆构建与关键帧检索从云端下沉至边缘,分两阶段运行:在摄入阶段,通过场景分割与聚类持续处理边缘流式视频,选择关键帧并用多模态嵌入模型生成分层记忆,实现高效存储与检索;在查询阶段,对输入查询进行记忆索引,并采用基于阈值的渐进采样算法选择关键帧,增强多样性,自适应平衡系统成本与推理准确率。大量评估表明,Venus相较现有最优方法在总响应延迟上提升15至131倍,可在数秒内实现实时响应,同时保持甚至超越原有推理精度。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have demonstrated impressive multimodal comprehension capabilities and are being deployed in an increasing number of online video understanding applications. While recent efforts extensively explore advancing VLMs' reasoning power in these cases, deployment constraints are overlooked, leading to overwhelming system overhead in real-world deployments. To address that, we propose Venus, an on-device memory-and-retrieval system for efficient online video understanding. Venus proposes an edge-cloud disaggregated architecture that sinks memory construction and keyframe retrieval from cloud to edge, operating in two stages. In the ingestion stage, Venus continuously processes streaming edge videos via scene segmentation and clustering, where the selected keyframes are embedded with a multimodal embedding model to build a hierarchical memory for efficient storage and retrieval. In the querying stage, Venus indexes incoming queries from memory, and employs a threshold-based progressive sampling algorithm for keyframe selection that enhances diversity and adaptively balances system cost and reasoning accuracy. Our extensive evaluation shows that Venus achieves a 15x-131x speedup in total response latency compared to state-of-the-art methods, enabling real-time responses within seconds while maintaining comparable or even superior reasoning accuracy.

视频理解边缘计算记忆系统多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。