arXiv:2606.19341cs.CVcs.CL2026-06中稿 · ICML被引 2

让视频理解像对话一样主动提问,按需看内容,省时又高效。

Native Active Perception as Reasoning for Omni-Modal Understanding

论文配图:Native Active Perception as Reasoning for Omni-Modal Understanding
图 1 · 摘自论文原文
  • 把视频理解变成边看边想的问答循环,只关注关键片段。
  • 在多个评测中超越开源模型,7B小模型胜过10倍大的大模型。
  • 适合需要长视频分析但算力有限的研究者和开发者。

传统被动模型处理长视频时需全量观看,计算成本随时长线性增长。现有交互式框架常依赖全局预扫描,上下文开销仍与视频长度相关。本文提出OmniAgent,首个原生多模态主动感知代理,将视频理解建模为基于部分可观测马尔可夫决策过程(POMDP)的观察-思考-行动迭代循环。该模型通过按需执行动作,选择性地从音视频中提炼信息并存入持久文本记忆,实现推理复杂度与原始视频时长解耦。为此,我们引入:(1) 代理监督微调,通过最佳N轨迹合成与双阶段质量控制启动原生主动感知;(2) 代理强化学习结合TAURA(转数感知自适应不确定性重标优势),利用每轮熵值引导信用分配至关键发现节点。重要的是,OmniAgent展现出正向测试时缩放特性——推理轮次越多,性能越强,验证了主动感知的有效性。在十个基准测试(如VideoMME、LVBench)上的实证结果表明,OmniAgent在开源模型中达到领先水平。尤其在LVBench上,7B版本性能达50.5%,超越10倍更大的Qwen2.5-VL-72B(47.3%)。

原文摘要 · Abstract (English)

Passive models for long video understanding typically rely on a "watch-it-all" paradigm, processing frames uniformly regardless of query difficulty, causing computational cost to grow with video duration. Although interactive frameworks have emerged, they often rely on global pre-scanning, and their context cost still scales with video length. We propose OmniAgent, the first native omni-modal agent that formulates video understanding as a POMDP-based iterative Observation-Thought-Action cycle. OmniAgent executes on-demand actions to selectively distill audio-visual cues into a persistent textual memory, effectively decoupling reasoning complexity from raw video duration. To operationalize this, we introduce (1) Agentic Supervised Fine-Tuning to bootstrap native active perception via best-of-N trajectory synthesis with dual-stage quality control, and (2) Agentic Reinforcement Learning with TAURA (Turn-aware Adaptive Uncertainty Rescaled Advantage), which leverages turn-level entropy to steer credit assignment toward pivotal discovery turns. Crucially, OmniAgent exhibits positive test-time scaling, where performance improves as the number of reasoning turns increases, validating the efficacy of active perception. Empirical results across ten benchmarks (e.g., VideoMME, LVBench) demonstrate that OmniAgent achieves state-of-the-art performance among open-source models. Notably, on LVBench, our 7B agent outperforms the 10$\times$ larger Qwen2.5-VL-72B (50.5% vs. 47.3%).

主动感知多模态推理增强视频理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。