让长视频理解模型在推理时动态重采样，提升关键帧保留能力。
LongVU-TTT: Causal Test-Time Training for Visual Resampling in Long Video Understanding
- 引入因果快速权重的卷积重采样器，实时调整视频特征。
- 在MLVU上比TTT-MLP提升2.12，比双向Mamba2提升3.04。
- 适合需要处理超长视频的视觉语言模型研究者。
长视频多模态大模型需在有限视觉令牌预算下建模时间变化，而多数帧信息已被压缩丢失。本文提出LongVU-TTT，通过在视觉编码器与大语言模型间插入卷积测试时训练（TTT）重采样器，实现因果快速权重更新。其分组2D快速权重能自适应每段视频，在压缩前对帧特征进行上下文建模；混合均匀性与变化感知选择器则显式保留关键帧证据以支持下游推理。在控制条件下，TTT-Conv在MLVU上较TTT-MLP提升最高达+2.12，较双向Mamba2提升最高达+3.04，且优于注意力与固定状态循环重采样器在三个基准上的表现。分析表明，快速权重更像时间聚合状态而非可靠长期记忆：其增益随证据距离增加而衰减，因此显式帧保留至关重要。LongVU-TTT可处理最多512帧，压缩至128个LLM可用帧，在五个视频理解基准上均取得有竞争力的表现。
原文摘要 · Abstract (English)
Long-video MLLMs must model temporal change before a limited visual-token budget removes most frame evidence. We introduce LongVU-TTT, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM. Its grouped 2D fast weights adapt to each video and contextualize frame features before compression, while a hybrid uniform-and-change-aware selector retains explicit visual evidence for downstream reasoning. Under controlled conditions, TTT-Conv improves over TTT-MLP by up to +2.12 and bidirectional Mamba2 by up to +3.04 on MLVU, and it is stronger than attention- and fixed-state recurrent resamplers across three benchmarks. Analysis shows that the fast weights behave as a temporal aggregation state rather than a reliable long-horizon episodic memory: their benefit attenuates as evidence becomes more distant, motivating explicit frame retention. LongVU-TTT processes up to 512 frames before reducing them to 128 LLM frames and achieves competitive performance across five video understanding benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。