用脑启发的混合记忆机制，让视频理解持续不丢细节。
FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding
- 分频域与空间缩略图双模块，兼顾短期清晰与长期连贯
- 在三个基准上分别提升5.20%、4.52%、2.34%性能
- 无需训练即可部署，适合长时视频流场景
将多模态大语言模型从离线转向在线流式视频理解，对实现持续感知至关重要。然而现有方法缺乏灵活适应性，导致细节不可逆丢失与上下文碎片化。为此，我们提出受大脑对数感知与记忆固化启发的频率空间混合记忆网络 FreshMem。FreshMem 通过两个协同模块，实现短期保真与长期连贯的平衡：多尺度频域记忆（MFM）将溢出帧投影为代表性频域系数，并辅以残差细节以重构全局历史“概要”；空间缩略图记忆（STM）则采用自适应压缩策略，将连续流划分为事件片段，并提炼为高密度空间缩略图。大量实验表明，FreshMem 显著提升 Qwen2-VL 基线，在 StreamingBench、OV-Bench 与 OVO-Bench 上分别获得 5.20%、4.52% 和 2.34% 的性能增益。作为免训练方案，FreshMem 超越多个全微调方法，为长时程流式视频理解提供高效范式。
原文摘要 · Abstract (English)
Transitioning Multimodal Large Language Models (MLLMs) from offline to online streaming video understanding is essential for continuous perception. However, existing methods lack flexible adaptivity, leading to irreversible detail loss and context fragmentation. To resolve this, we propose FreshMem, a Frequency-Space Hybrid Memory network inspired by the brain's logarithmic perception and memory consolidation. FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules: Multi-scale Frequency Memory (MFM), which projects overflowing frames into representative frequency coefficients, complemented by residual details to reconstruct a global historical "gist"; and Space Thumbnail Memory (STM), which discretizes the continuous stream into episodic clusters by employing an adaptive compression strategy to distill them into high-density space thumbnails. Extensive experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively. As a training-free solution, FreshMem outperforms several fully fine-tuned methods, offering a highly efficient paradigm for long-horizon streaming video understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。