用固定大小内存存关键视频片段，提速18到75倍。
Long-VMNet: Accelerating Long-Form Video Understanding via Fixed Memory
- 用神经采样器提取视频中的关键帧，存入固定大小内存。
- 单次扫描视频即可完成理解，推理速度提升18至75倍。
- 适合需要快速处理长视频的场景，如检索与问答。
长视频理解在视频检索、摘要和问答等应用中至关重要，但传统方法计算开销大，常受GPU内存限制。为此，我们提出长视频记忆网络（Long-VMNet），通过固定大小的记忆表示存储从输入视频中采样的判别性片段。Long-VMNet利用神经采样器识别判别性标记，仅需一次视频扫描即可实现高效处理。在Rest-ADL数据集上的实验表明，该方法在长视频检索与问答任务中推理时间缩短了18至75倍，同时保持了具有竞争力的预测性能。
原文摘要 · Abstract (English)
Long-form video understanding is essential for various applications such as video retrieval, summarizing, and question answering. Yet, traditional approaches demand substantial computing power and are often bottlenecked by GPU memory. To tackle this issue, we present Long-Video Memory Network, Long-VMNet, a novel video understanding method that employs a fixed-size memory representation to store discriminative patches sampled from the input video. Long-VMNet achieves improved efficiency by leveraging a neural sampler that identifies discriminative tokens. Additionally, Long-VMNet only needs one scan through the video, greatly boosting efficiency. Our results on the Rest-ADL dataset demonstrate an 18x -- 75x improvement in inference times for long-form video retrieval and answering questions, with a competitive predictive performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。