arXiv:2606.11913cs.CV2026-06

将长视频转化为可复用的知识权重,实现秒级问答。

From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations

论文配图:From Content to Knowledge: Lightning Fast Long-Video Understanding with Neural Knowledge Representations
图 1 · 摘自论文原文
  • 用神经知识权重封装视频语义,替代传统编码方式。
  • 一次训练后推理延迟降低超99%,性能媲美顶尖方法。
  • 适合需要多次交互式问答的长视频应用。

我们提出一种新范式:将长视频视为神经知识表示(NKR)。NKR不以文本流或预组织数据库形式存储内容,而是作为附加于视觉-语言模型(VLM)主干的少量网络权重。通过新型代理知识蒸馏(AKD)过程,一个智能体自动生成密集描述与问答对,将视频知识蒸馏至NKR中。AKD为一次性完整编码阶段,完成后得到的NKR成为可移植、可复用的资产。推理时仅需将轻量级NKR挂载到冻结的VLM上,即可实现无需重新加载或编码原视频的直接查询理解。该方法使视频长度与推理成本解耦,显著提升多轮交互式理解的平均效率。在LVBench基准测试中,本方法性能接近当前最优水平,同时端到端延迟降低超过两个数量级,为交互式长视频理解开辟新可能。

原文摘要 · Abstract (English)

We propose a new paradigm for long video understanding by treating a long video as a Neural Knowledge Representation (NKR). NKR represents video contents neither as a stream of tokens nor pre-organized databases, but as an individual small portion of network weights attached to the VLM backbone. The NKR weights are optimized to encapsulate the video's semantic content via a novel Agentic Knowledge Distillation (AKD) process, where an agent automatically synthesizes dense descriptions and question-answer pairs to distill the video's knowledge into the NKR. While AKD serves as a comprehensive, one-time encoding phase, the resulting NKR transforms the video into a portable, reusable asset. At inference, the lightweight NKR is mounted onto a frozen Vision-Language Model (VLM), enabling direct, query-based understanding without reloading or re-encoding the original video. This approach decouples video length from inference cost, offering high amortized efficiency for multi-turn video understanding. Experiments on the LVBench benchmark show our method achieves performance comparable to state-of-the-art approaches while reducing end-to-end latency by over two orders of magnitude, opening new possibilities for interactive long-video understanding.

长视频理解知识蒸馏VLM高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。