让视频工具并行调用,提升大模型视频理解的效率与鲁棒性。
ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning
- 设计并行工具调用框架,单轮可执行多个视频裁剪操作。
- 在六个基准上平均提升7.9%,格式合规率从0.13升至0.64。
- 解决工具先验导致的训练崩溃问题,适合研究智能体强化学习者。
通过强化学习(RL)训练大型多模态模型(LMMs)原生调用视频处理工具(如裁剪),已成为实现长视频理解的有前景路径。然而,现有方法仅支持逐轮调用(每轮一个操作),易导致错误传播、上下文污染,且推理成本随轮次线性增长。本文提出ParaVT,首个支持并行视频工具调用的多智能体端到端强化学习框架,可在单轮中同时调度多个时间窗裁剪,保持上下文清晰并增强容错能力。但标准强化学习在该框架下遭遇‘工具先验悖论’:预训练工具先验虽促进探索,却也破坏结构格式,并暴露跳过工具调用的奖励捷径。跨模型对比实验表明,弱先验模型格式稳定但零调用,证实先验强度是格式坍塌与工具探索的共因。为此,提出PARA-GRPO算法,引入两项机制:(i) 仅在易崩溃结构标记位置施加格式奖励;(ii) 每提示随机分配帧预算,确保调用工具能获得可观测回报。在六项长视频理解基准上,ParaVT相较Qwen3-VL基线平均提升7.9%;PARA-GRPO将训练期格式合规率从0.13提升至0.64。随着工具能力内化于现代LMMs,强化学习需协同应对先验影响,ParaVT提供通用解决方案。代码、数据与模型权重已公开。
原文摘要 · Abstract (English)
Training large multimodal models (LMMs) via reinforcement learning (RL) to natively invoke video-processing tools (e.g., cropping) has become a promising route to long-video understanding. However, existing native-RL methods dispatch tool calls sequentially (i.e., one per turn): a single wrong crop propagates errors without peer correction, multi-turn tool calls corrupt context, and inference cost scales linearly with the number of turns. We introduce ParaVT, the first multi-agent end-to-end RL-trained framework for Parallel Video Tool calling, dispatching multiple time-window crops in a single turn for cleaner context and better fault tolerance. Yet applying standard RL to ParaVT reveals an obstacle we term the Tool Prior Paradox: the pretrained tool priors that enable tool exploration also destabilize cold-started structural format and expose the skip-tool reward shortcut under temperature sampling. A cross-model contrast on a weaker-prior LMM supports this claim: format stays stable but RL elicits zero tool calls, indicating that prior strength is the shared driver of both format collapse and tool exploration. We propose PARA-GRPO (Parseability-Anchored and Ratio-gAted GRPO), which augments standard RL with two complementary mechanisms: (i) a targeted format reward applied only at the structural-token positions most prone to collapse, and (ii) a per-prompt frame-budget randomization that creates training prompts where calling the tool yields a measurable reward signal over skipping it. Across six long-video understanding benchmarks, ParaVT improves over the Qwen3-VL baseline by +7.9% on average, with PARA-GRPO lifting training-time format compliance from 0.13 to 0.64. As tool capabilities become increasingly internalized in modern LMMs, RL must cooperate with the resulting priors, and ParaVT offers a general recipe for agentic RL. Code, data, and model weights are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。