让大模型学会按需放大长视频片段,精准定位关键信息。
OmniReasoner: Thinking with Long Audio-Video via Native Tool Use

- 通过工具调用机制,动态决定何时放大局部视频音频
- 在多个基准测试中提升准确率与时间定位精度
- 适合需要处理长时多模态内容的智能分析场景
长时多模态推理对通用多模态大模型而言极具挑战,因关键证据往往稀疏、跨模态且难以以高保真度统一保存。我们提出OmniReasoner,一种面向长时音视频推理的工具使用后训练框架:多模态大模型通过监督微调和强化学习,学会在回答前判断是否及何处调用「放大」工具。该方法先构建全视频低成本全局预览,必要时请求指定时间区间,调用放大工具进行高保真视听检查。由于模型在调用前后观察到不同采样粒度——稀疏全局预览与密集局部片段——我们引入TimeAnchor机制,确保工具的时间参数在不同粒度间保持有效且往返一致,而非绑定于特定采样率的帧索引。为避免昂贵的人工标注间隔,我们设计了时间增强数据引擎,通过视频编辑合成工具使用后训练轨迹。跨多模态与视频基准测试表明,OmniReasoner在提升答案准确率与时间定位能力的同时,将高保真计算集中在信息丰富区域。代码已开源:https://github.com/RockyChen0205/OmniReasoner。
原文摘要 · Abstract (English)
Long audio-video reasoning is difficult for omnimodal LLMs because the decisive evidence is often sparse, cross-modal, and too expensive to preserve with uniformly high-fidelity inputs. We introduce OmniReasoner, a tool-use post-training framework for Thinking with Long Audio-Video: omni-modal LLMs learn, via supervised fine-tuning and reinforcement learning, to decide whether and where to call a zoom-in tool before answering. OmniReasoner first builds a low-cost global preview of the full stream and then, when needed, calls the zoom-in tool with a requested temporal interval for higher-fidelity visual and audio inspection before answering. Because the model observes different sampling granularities before and after this call -- a sparse global preview and a denser local clip -- we introduce TimeAnchor, which keeps the tool's temporal argument valid and round-trip-consistent across these granularities, rather than tied to frame indices from a particular sampling rate. To make this tool-use behavior trainable without expensive manual interval annotation, we build a Temporal Augmented Data Engine that synthesizes tool-use post-training trajectories by video editing and composition. Experiments across omnimodal and video benchmarks show that OmniReasoner improves both answer accuracy and temporal grounding while concentrating high-fidelity computation on informative regions. Code is available at https://github.com/RockyChen0205/OmniReasoner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。