arXiv:2606.09447cs.AI2026-06

用蒸馏+强化学习训练低成本云控代理,自动验证文档与真实界面一致性。

AliyunConsoleAgent: Training Web Agents in Real-World Cloud Environments via Distillation and Reinforcement Learning

论文配图:AliyunConsoleAgent: Training Web Agents in Real-World Cloud Environments via Distillation and Reinforcement Learning
图 1 · 摘自论文原文
  • 先用大模型轨迹蒸馏微调,再在真实云环境用强化学习优化决策。
  • 在278项任务中达63.52%成功率,比基线提升20.24个百分点。
  • 成本降低92%,适合大规模部署于企业级云平台运维场景。

我们提出AliyunConsoleAgent,一个用于真实云控制台自动化文档验证的网页代理框架。主要云平台包含数百个产品且功能迭代频繁,导致控制台界面与文档常出现偏差。每年需约400万次重复检查以确保文档可端到端执行,但人工覆盖率不足1%。虽前沿大模型代理成功率高,但其高昂成本和数据隐私问题限制了大规模应用。我们采用两阶段训练:先在蒸馏的大模型轨迹上进行监督微调,再在真实云环境中使用组相对策略优化(GRPO)和双通道结果奖励模型进行强化学习。为支持大规模强化学习,构建了基于Terraform资源预置、大模型驱动按需创建的高确定性回放系统,有效隔离环境噪声。同时引入基于后端审计日志的规则化奖励评估协议,实现客观、抗奖励欺骗的结果判断。模型从机械执行指令进化为具备云产品特定理解的自主决策能力。在包含278个任务的挑战性基准测试中,最佳前沿模型仅达65.34%成功率,而AliyunConsoleAgent-32B达到63.52%平均成功率,较基线提升20.24个百分点,与顶尖模型差距缩小至1.82个百分点(置信区间[-1.27, 7.39]),推理成本降低92%。

原文摘要 · Abstract (English)

We present AliyunConsoleAgent, a web agent framework for automated documentation verification in real-world cloud consoles. Major cloud platforms encompass hundreds of products with rapid feature iteration, causing console UIs to frequently diverge from their corresponding documentation. Verifying that documented procedures accurately reflect the current console and can be executed end-to-end demands an estimated 4 million recurring inspections annually, yet manual coverage remains below 1%. While agent systems built on frontier proprietary models achieve high success rates, their prohibitive cost and data privacy constraints preclude large-scale deployment. We propose a two-stage training paradigm: supervised fine-tuning (SFT) on distilled frontier-model trajectories, followed by reinforcement learning using Group Relative Policy Optimization (GRPO) and a dual-channel outcome reward model in real cloud environments. To support large-scale RL training, we construct a high-determinism rollout system featuring Terraform-based resource pre-provisioning and LLM-driven on-demand provisioning, which effectively isolates environment noise from the training signal. We further introduce a rule-based reward evaluation protocol grounded in backend audit logs, providing objective, reward-hacking-resistant outcome judgment. Our model evolves from mechanical instruction following to autonomous decision-making with cloud console and product-specific understanding. Experiments on a challenging 278-task benchmark where the best frontier model achieves only 65.34% demonstrate that AliyunConsoleAgent-32B achieves a 63.52% mean success rate -- a 20.24 percentage-point improvement over the base model, narrowing the gap to the best frontier proprietary model to 1.82 pp (bootstrap 95% CI [-1.27, 7.39]) -- at 92% lower inference cost.

云智能强化学习自动化验证大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。