用现成多模态大模型实现高效在线记忆问答，内存占用降万倍
How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?
- 将视频流转为每分钟仅几KB的文本记忆
- 在封闭数据集上达56.0%准确率，内存效率提升10万倍
- 无需额外训练，适合资源受限场景应用
我们研究了现成的多模态大模型（MLLMs）是否能在不进行额外训练的情况下完成在线情景记忆视频问答（OEM-VQA）。我们的流程通过MLLM描述模块将连续的视角视频转化为轻量级文本记忆，每分钟仅约3.6 KB；再由LLM推理模块查询该记忆以回答多项选择题。在QAEgo4D-Closed基准上，最优配置达到56.0%的准确率，性能媲美专用顶尖系统，同时内存效率提升10⁴至10⁵倍。大量消融实验揭示了各组件的作用与设计影响，也为未来研究指明改进方向。
原文摘要 · Abstract (English)
We investigate whether off-the-shelf Multimodal Large Language Models (MLLMs) can tackle Online Episodic-Memory Video Question Answering (OEM-VQA) without additional training. Our pipeline converts a streaming egocentric video into a lightweight textual memory, only a few kilobytes per minute, via an MLLM descriptor module, and answers multiple-choice questions by querying this memory with an LLM reasoner module. On the QAEgo4D-Closed benchmark, our best configuration attains 56.0% accuracy with 3.6 kB per minute storage, matching the performance of dedicated state-of-the-art systems while being 10**4/10**5 times more memory-efficient. Extensive ablations provides insights into the role of each component and design choice, and highlight directions of improvement for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。