arXiv:2601.06301cs.CRcs.AI2026-01被引 2

LLM让普通人也能轻松抓取复杂网页数据,只需一句指令。

Beyond BeautifulSoup: Benchmarking LLM-Powered Web Scraping for Everyday Users

  • 用自然语言指令让LLM自动生成或自主执行抓取流程。
  • 端到端代理仅需少于5次修改即可完成复杂网站抓取。
  • 适合无技术背景用户,也揭示了潜在安全风险。

网页抓取传统上需要掌握HTML解析、会话管理与反反爬技巧,限制了非专业开发者的大规模数据获取。我们提出,大语言模型(LLMs)已使这一过程民主化,让低技能用户通过简单自然语言提示完成复杂操作。尽管现有评测多在专家理想条件下进行,我们系统评估了普通用户使用现成LLM工具在35个网站上的表现,覆盖五类安全层级(含认证、反机器人与验证码)。设计并测试两种工作流:(a) LLM辅助脚本,用户生成代码但手动执行;(b) 端到端LLM智能体,通过集成工具自主导航与提取数据。结果显示,端到端代理已使复杂抓取变得可及——仅需单次提示且少于5次调整即可完成任务。同时发现,对静态站点,LLM辅助脚本可能更简单高效。基于此,我们为新手提供操作指南,并分析潜在攻击者利用这些技术的可能性。

原文摘要 · Abstract (English)

Web scraping has historically required technical expertise in HTML parsing, session management, and authentication circumvention, which limited large-scale data extraction to skilled developers. We argue that large language models (LLMs) have democratized web scraping, enabling low-skill users to execute sophisticated operations through simple natural language prompts. While extensive benchmarks evaluate these tools under optimal expert conditions, we show that without extensive manual effort, current LLM-based workflows allow novice users to scrape complex websites that would otherwise be inaccessible. We systematically benchmark what everyday users can do with off-the-shelf LLM tools across 35 sites spanning five security tiers, including authentication, anti-bot, and CAPTCHA controls. We devise and evaluate two distinct workflows: (a) LLM-assisted scripting, where users prompt LLMs to generate traditional scraping code but maintain manual execution control, and (b) end-to-end LLM agents, which autonomously navigate and extract data through integrated tool use. Our results demonstrate that end-to-end agents have made complex scraping accessible - requiring as little as a single prompt with minimal refinement (less than 5 changes) to complete workflows. We also highlight scenarios where LLM-assisted scripting may be simpler and faster for static sites. In light of these findings, we provide simple procedures for novices to use these workflows and gauge what adversaries could achieve using these.

网页抓取LLM应用自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。