动态调整视频帧采样密度,让模型更高效地找证据、做推理。
DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding

- 将时间窗口和采样密度作为可学习的令牌,在一次生成中完成多粒度证据获取。
- 在六个基准上表现优于7B-8B模型,8B版本在多数指标创最佳纪录。
- 适合需要精准视频理解与高效推理的多模态大模型研究者使用。
当前视频多模态大模型虽能结合分步推理与按需视觉证据检索,但在结构上仍存在两大缺陷:(i) 帧采样密度无法学习,固定帧率导致细粒度证据需多次检索,增加上下文长度与训练难度;(ii) 检索与回答优化共享单一轨迹级奖励,无法区分‘看哪里’和‘怎么答’的贡献。为此,我们提出DynFrame框架,将时间窗口与采样密度作为自回归生成中的原生令牌,实现单次检索获取多粒度证据。基于此令牌化接口,进一步设计分段解耦的GRPO(SD-GRPO),在检索边界分割每条轨迹,分别赋予角色特定的令牌级奖励,独立评估采样决策与答案生成。在DM-CoT-74k和DM-RL-45k数据集上训练的DynFrame-4B,在六个基准(NExT-GQA、Charades-STA、ActivityNet-MR、Video-MME、MLVU、LVBench)上表现媲美7B-8B强基线,而DynFrame-8B在多数指标上刷新最优记录。代码已开源。
原文摘要 · Abstract (English)
Recent video multimodal large language models (MLLMs) increasingly couple step-by-step reasoning with on-demand visual evidence retrieval, allowing models to revisit relevant video segments during inference. However, two structural gaps remain in existing thinking-with-video systems. (i) Sampling density is not a learnable decision: existing methods may let the model decide where to look, but the per-window frame rate is largely fixed. As a result, fine-grained evidence is often recovered through repeated retrieval calls, which increases inference context length and training difficulty. (ii) Retrieval and answer generation are usually optimized with a single trajectory-level advantage, so the "where to look" tokens and the "how to answer" tokens receive the same credit even when one is correct and the other is not. To address these gaps, we present DynFrame, a framework that emits the temporal window and the sampling density as native tokens within a single autoregressive pass. This learnable span-density retrieval enables acquiring multi-granularity evidence with a single retrieval step. Based on the above tokenized retrieval interface, we further introduce Segment-Decoupled GRPO (SD-GRPO), which splits each rollout at the retrieval boundary and assigns role-specific token-level advantages, separately crediting the sampling decision and the answer. Trained on the curated DM-CoT-74k and DM-RL-45k, DynFrame-4B is competitive with strong 7B-8B baselines across six benchmarks (NExT-GQA, Charades-STA, ActivityNet-MR, Video-MME, MLVU, LVBench), and DynFrame-8B sets new state-of-the-art on most metrics. Code is available at https://github.com/zhangguanghao523/DynFrame.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。