用分层编码提升长视频理解，减少计算量同时增强运动感知。
HY-Himmel Technical Report: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding
- 分两路处理：关键帧由大模型精算，中间帧用轻量适配器提取运动特征
- 在Video-MME上比密集32帧基线高2.3个百分点，上下文令牌减少3.6倍
- 适合需要高效处理长视频的多模态应用，如视频问答与内容分析
多模态语言模型在长视频理解中面临三大瓶颈：密集RGB帧生成导致解码开销大，帧数增加引发令牌数量二次增长，以及稀疏关键帧采样下运动感知弱。本文提出HY-Himmel框架，通过分层设计将语义与运动能力分离：少量稀疏锚点I帧送入昂贵的ViT模型以确定物体身份和场景布局；而更密集的帧间间隔则由轻量级压缩域三流适配器编码，该适配器从运动矢量图、残差图和I帧上下文提炼运动证据，生成对齐的运动令牌。这些令牌经可微占位符机制注入LLM前，先经过专用阶段一对比对齐，确保运动表征与冻结视觉主干兼容。在Video-MME测试中，相比32帧密集基线，性能提升2.3个百分点（61.2% → 63.5%），同时使用上下文令牌减少3.6倍。大量消融实验验证三流结构为性能提升的必要且充分条件。
原文摘要 · Abstract (English)
Long-video understanding with multimodal language models suffers from three compounding bottlenecks: heavy decode cost to obtain dense RGB frames, quadratic token growth with frame count, and weak motion perception under sparse keyframe sampling. We present HY-Himmel, a hierarchical video-language framework that allocates semantic and motion capacity separately. A small set of sparse anchor I-frames is routed to the expensive host ViT to ground object identity and scene layout, while the far denser inter-frame intervals are encoded by a lightweight compressed-domain tri-stream adapter that distils motion evidence from motion-vector maps, residual maps, and I-frame context into aligned motion tokens. These tokens are injected into the LLM via a differentiable placeholder mechanism after a dedicated Stage-1 contrastive alignment that places the motion representation in a geometry compatible with the frozen visual backbone. On Video-MME, HY-Himmel surpasses the dense 32-frame baseline by +2.3 pp (61.2 to 63.5%) while using 3.6x fewer context tokens. Extensive ablations over stream composition, motion encoder family, fusion mode, alignment objective, anchor count, LoRA rank, and video duration confirm that the full tri-stream is necessary and sufficient for the observed gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。