用边缘设备实时问答记忆视频，兼顾隐私与速度。
Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge
- 分两线程处理：一边流式转视频为文本记忆，一边用文本回答问题。
- 8GB消费级显卡上达51.76%准确率，首字耗时仅0.41秒。
- 结果媲美云端，适合对隐私敏感的可穿戴设备场景。
我们研究了在边缘设备上使用多模态大语言模型（MLLMs）进行实时在线情景记忆问答的可行性。尽管云部署常见，但其带来隐私和延迟问题，尤其对可穿戴助手而言。因此，我们探索在边缘端实现。将流式约束融入问答流程，采用双异步线程结构：描述线程持续将视频转为轻量文本记忆，问答线程基于文本记忆推理回答查询。在QAEgo4D-Closed基准上评估了在严格资源限制下MLLM的表现，结果显示在性能上也优于云方案。具体地，在消费级8GB GPU上实现端到端配置，准确率达51.76%，首字延迟（TTFT）为0.41秒；扩展至本地企业级服务器，准确率提升至54.40%，TTFT为0.88秒。相比之下，云方案准确率为56.00%。这些有竞争力的结果凸显了边缘方案在保护隐私的情景记忆检索中的潜力。
原文摘要 · Abstract (English)
We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering. While cloud offloading is common, it raises privacy and latency concerns for wearable assistants, hence we investigate implementation on the edge. We integrated streaming constraints into our question answering pipeline, which is structured into two asynchronous threads: a Descriptor Thread that continuously converts video into a lightweight textual memory, and a Question Answering (QA) Thread that reasons over the textual memory to answer queries. Experiments on the QAEgo4D-Closed benchmark analyze the performance of Multimodal Large Language Models (MLLMs) within strict resource boundaries, showing promising results also when compared to clound-based solutions. Specifically, an end-to-end configuration running on a consumer-grade 8GB GPU achieves 51.76% accuracy with a Time-To-First-Token (TTFT) of 0.41s. Scaling to a local enterprise-grade server yields 54.40% accuracy with a TTFT of 0.88s. In comparison, a cloud-based solution obtains an accuracy of 56.00%. These competitive results highlight the potential of edge-based solutions for privacy-preserving episodic memory retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。