arXiv:2606.04351cs.CVcs.CL2026-06

用少量参数将视频压缩成可快速调用的适配器,大幅降低视觉模型推理开销。

Frames2LoRA: Parametric Video Internalization for Vision-Language Models

论文配图:Frames2LoRA: Parametric Video Internalization for Vision-Language Models
图 1 · 摘自论文原文
  • 通过超网络直接生成视频对应的LoRA适配器,无需迭代训练。
  • 在多个基准上性能接近原生视频输入,且推理耗时降低6-80倍。
  • 支持长视频分段处理,适合需要高效推理的视频理解场景。

在视觉语言模型中处理视频成本高昂:每帧占用数百个标记,推理代价随帧数和重复查询线性增长。我们提出Frames2LoRA,一种参数化视频内化方法。一个感知器超网络读取冻结的VLM编码视频时逐层产生的中间表示,并在单次前向传播中生成低秩适配器(LoRA)。与标准LoRA微调不同,该方法直接从视频预测权重,无需梯度更新。在SmolVLM2 500M和2.2B模型上,针对视频摘要与字幕任务训练后,相同的冻结VLM仅通过适配器即可回答问题,查询时上下文零视觉标记。在所有五个字幕基准上,其性能统计上不劣于直接视频输入,且在七组八对视频问答任务中表现相当。尽管仅在12帧、384px下训练,仍可稳定处理高达1,024帧、1024px的视频,而直接视频输入常出现退化。在整个测试范围内,答案阶段视觉标记量减少最多达1,500倍,查询首字延迟(TTFT)降低6-80倍,同时保持视频忠实输出。此外,我们发现非重叠视频片段独立生成的适配器可在秩空间中组合,为长视频分块内化提供路径。

原文摘要 · Abstract (English)

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.

视频理解LoRA高效推理参数化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。