arXiv:2606.01599cs.AI2026-06被引 3

TRON生成可验证的在线视觉推理环境,支持动态训练与能力分级评估。

TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RL

  • 基于可控生成器-验证器程序实时构造新场景,按需生成无限量训练样本。
  • 在520个环境上测试,模型在10个外部多模态推理基准上性能显著提升。
  • 适合需要持续学习与可验证训练信号的视觉推理强化学习研究者。

视觉推理强化学习需要可扩展、可验证且可控的训练信号。现有方法依赖静态预收集数据集,样本数量受限于采集预算。本文提出TRON(Targeted, Rule-verifiable Online eNvironments),一种在线环境底座:通过可控生成-验证程序按需生成训练轨迹,采样潜在视觉状态,渲染图像,提出问题,并精确验证答案。单次运行即可按当前课程难度生成无限量新鲜实例。当前TRON包含520个环境,分为五个能力桶(空间、数学、图表、模式/逻辑、计数);同一底座支持全能力通用模型与分能力专精模型,无需额外数据收集。我们还进行了底座分析,涵盖生成可靠性、实例与难度多样性、跨环境近似重复及基础模型在不同难度下的通过率。使用METHOD进行后训练,显著提升了Qwen3-VL-4B、Qwen2.5-VL-7B和MiMo-VL-7B-SFT在十个外部多模态推理基准上的表现。

原文摘要 · Abstract (English)

Reinforcement learning (RL) for visual reasoning needs scalable, verifiable, and controllable training signals. Existing visual RL post-training trains on static curated datasets, with fixed image-question-answer samples bounded by their collection budget. In this work, we introduce TRON (Targeted, Rule-verifiable Online eNvironments), an online environment substrate: a training rollout is generated on demand by a controllable generator-verifier program that samples a fresh latent visual state, renders an image, asks a question, and exactly verifies the answer. A single run can therefore draw an unbounded stream of fresh instances at the difficulty level required by the current curriculum. The current TRON suite contains 520 environments organized into five ability buckets (spatial, mathematical, diagram, pattern/logic, and counting); the same substrate supports both a single full model trained on all buckets and per-bucket ability-specialist models, with no additional data collection. We also introduce a substrate analysis covering generation reliability, instance and level diversity, cross-environment near-duplicates, and base-model pass rate by difficulty level. RL post-training with METHOD consistently improves performance on ten external multimodal reasoning benchmarks across Qwen3-VL-4B, Qwen2.5-VL-7B, and MiMo-VL-7B-SFT.

强化学习视觉推理在线训练可验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。