用记忆树结构加速3D场景问答，提升效率与准确率
Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering
- 构建可实时更新的3D记忆树，按需检索关键帧
- 在OpenEQA上使GPT-4o准确率提升17.4%、LLaVA提升5.8%
- 适合资源受限的机器人或虚拟助手场景使用
在具身场景中高效准确地回答问题面临计算与内存资源有限的挑战。现有方法采用视觉搜索从数千帧视频中检索关键帧，但效率低下。本文提出一种基于记忆树的键帧选择范式，利用轻量且可复用的3D场景表示MemTree3D，支持基于相机6-DoF位姿的实时在线构建。MemTree3D捕捉多层级3D场景信息，使大语言模型通过评分机制高效查询并检索与问题相关的关键帧，无需重新处理整个视频流。在OpenEQA数据集上，该方法使GPT-4o的LLM-Match提升17.4%，LLaVA-OneVision-7B提升5.8%，优于现有视觉搜索方法。代码已开源。
原文摘要 · Abstract (English)
Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。