通过强化学习提升多模态智能体的持续交互能力
PyVision-RL: Forging Open Agentic Vision Models via RL
- 采用采样-过滤-排序策略与累积工具奖励防止交互崩溃
- 视频任务中仅调用相关帧，视觉令牌消耗减少显著
- 适合构建高效、可扩展的开放权重多模态智能体
针对多模态智能体在强化学习中出现的交互崩溃问题（即模型减少工具使用和多轮推理），我们提出PyVision-RL框架，用于开放权重多模态模型。该框架结合过采样-过滤-排序的回放策略与累积工具奖励机制，稳定训练过程并维持持续交互。基于统一训练流程，我们构建了用于图像理解的PyVision-Image和用于视频理解的PyVision-Video。后者引入按需上下文构建机制，在推理时仅选择任务相关帧，大幅降低视觉令牌使用量。实验表明，该方法在性能和效率上均有显著提升，验证了持续交互与按需视觉处理对可扩展多模态智能体的关键作用。
原文摘要 · Abstract (English)
Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。