用训练中模型权重合并生成免费教师，提升视觉智能体训练效率
GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Training
- 通过合并训练过程中的模型权重，自动生成指导教师
- 准确率提升10-30%，训练时间减少50%，计算成本降低60%
- 无需依赖昂贵教师模型，适合资源受限的实践场景
基于视觉语言模型（VLM）的多轮强化学习在多模态智能体训练中面临奖励稀疏和长程信用分配难题。现有方法如引导思维强化（GTR）和在线策略蒸馏通过查询教师模型提供步骤级反馈来密化奖励，但依赖昂贵且常为特权的教师模型，限制了实用性与可复现性。本文提出GTR-Turbo，一种高效升级版GTR方法，在不训练或调用昂贵教师模型的前提下达到相同性能。具体地，GTR-Turbo将持续强化学习过程中产生的检查点权重合并，形成“免费”教师模型，通过监督微调或软对数蒸馏指导后续训练。该设计摆脱对特权VLM（如GPT、Gemini）的依赖，缓解了先前工作中观察到的“熵坍缩”问题，维持稳定训练。在多种视觉智能体任务上，相较于GTR，GTR-Turbo使基线模型准确率提升10-30%，训练时间减少50%，计算成本降低60%。
原文摘要 · Abstract (English)
Multi-turn reinforcement learning (RL) for multi-modal agents built upon vision-language models (VLMs) is hampered by sparse rewards and long-horizon credit assignment. Recent methods densify the reward by querying a teacher that provides step-level feedback, e.g., Guided Thought Reinforcement (GTR) and On-Policy Distillation, but rely on costly, often privileged models as the teacher, limiting practicality and reproducibility. We introduce GTR-Turbo, a highly efficient upgrade to GTR that matches its performance without training on or querying an expensive teacher model. Specifically, GTR-Turbo merges the weights of checkpoints produced during ongoing RL training and then uses the resulting merged model as a "free" teacher to guide subsequent RL via supervised fine-tuning or soft logit distillation. This design removes dependence on privileged VLMs (e.g., GPT or Gemini), mitigates the "entropy collapse" observed in prior work, and maintains stable training. Across diverse visual agentic tasks, GTR-Turbo improves the accuracy of the baseline model by 10-30% while reducing wall-clock training time by 50% and compute cost by 60% relative to GTR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。