用60亿令牌的网页工作流数据微调开源大模型，提升专业网页代理性能。
ScribeAgent: Towards Specialized Web Agents Using Production-Scale Workflow Data
- 用250个领域的真实网页操作数据微调开源LLM，增强对HTML等专业场景理解。
- 在Mind2Web上达最新水平，在WebArena上任务成功率提升7.3%。
- 适合研究网页自动化、Agent设计及低成本部署的开发者参考。
大型语言模型（LLM）代理正快速提升处理复杂网页任务的能力。现有方法多依赖通用闭源模型如GPT-4，并通过优化提示词来增强规划能力。然而，通用LLM缺乏对特定网页上下文（如HTML）的针对性训练，且在长时序规划中表现不佳。本文提出一种替代方案：使用来自250个领域、总计60亿令牌的生产级工作流数据，对开源LLM进行微调。该简单有效的方法在现有基准上显著优于基于提示的代理——ScribeAgent在Mind2Web上实现最先进的直接生成性能，在WebArena上比此前最优纯文本代理的任务成功率提高7.3%。我们还进行了详细的消融实验，分析了模型选择、训练策略、上下文窗口优化及数据集规模的影响，为实际应用提供可复现的指导。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents are rapidly improving to handle increasingly complex web-based tasks. Most of these agents rely on general-purpose, proprietary models like GPT-4 and focus on designing better prompts to improve their planning abilities. However, general-purpose LLMs are not specifically trained to understand specialized web contexts such as HTML, and they often struggle with long-horizon planning. We explore an alternative approach that fine-tunes open-source LLMs using production-scale workflow data collected from over 250 domains corresponding to 6 billion tokens. This simple yet effective approach shows substantial gains over prompting-based agents on existing benchmarks -- ScribeAgent achieves state-of-the-art direct generation performance on Mind2Web and improves the task success rate by 7.3% over the previous best text-only web agents on WebArena. We further perform detailed ablation studies on various fine-tuning design choices and provide insights into LLM selection, training recipes, context window optimization, and effect of dataset sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。