通过统计方法优化大模型网页代理训练，降低算力消耗并提升性能。
How to Train Your LLM Web Agent: A Statistical Diagnosis
- 采用师生模型两阶段训练：监督微调+在线强化学习。
- 仅用55%算力即达纯监督微调峰值表现，且超越闭源模型差距。
- 提供可复现的超参配置，避免无效试错，适合开源研究者参考。
基于大模型的网页代理近期取得显著进展，但主要集中在闭源系统，导致开源方案落后。核心挑战在于：一、多步网页交互任务关注不足；二、后训练所需算力过高。为此，我们首次开展面向大模型网页代理后训练的统计学研究。提出两阶段流程：用Llama 3.1 8B学生模仿Llama 3.3 70B教师进行监督微调（SFT），再进行在线强化学习（on-policy RL）。发现该过程对超参数极为敏感，穷举搜索不可行。因此，我们采样1,370种配置，利用自助法估计有效超参数。结果表明，结合SFT与在线强化学习在WorkArena和MiniWob++上均优于单一方法；且仅需55%算力即可达到纯SFT在MiniWob++上的最佳性能，显著推进算力-性能帕累托前沿，并唯一能缩小与闭源模型的差距。
原文摘要 · Abstract (English)
LLM-based web agents have recently made significant progress, but much of it has occurred in closed-source systems, widening the gap with open-source alternatives. Progress has been held back by two key challenges: first, a narrow focus on single-step tasks that overlooks the complexity of multi-step web interactions; and second, the high compute costs required to post-train LLM-based web agents. To address this, we present the first statistically grounded study on compute allocation for LLM web-agent post-training. Our approach uses a two-stage pipeline, training a Llama 3.1 8B student to imitate a Llama 3.3 70B teacher via supervised fine-tuning (SFT), followed by on-policy reinforcement learning. We find this process highly sensitive to hyperparameter choices, making exhaustive sweeps impractical. To spare others from expensive trial-and-error, we sample 1,370 configurations and use bootstrapping to estimate effective hyperparameters. Our results show that combining SFT with on-policy RL consistently outperforms either approach alone on both WorkArena and MiniWob++. Further, this strategy requires only 55% of the compute to match the peak performance of pure SFT on MiniWob++, effectively pushing the compute-performance Pareto frontier, and is the only strategy that can close the gap with closed-source models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。