用多模态大模型自动抓取动态网页，解决传统爬虫失效问题。
Webscraper: Leverage Multimodal Large Language Models for Index-Content Web Scraping
- 基于多模态大模型的五阶段提示框架，自主导航交互页面。
- 在6个新闻网站上准确率显著优于基线模型Anthropic Computer Use。
- 可拓展至电商等场景，适合需要动态数据提取的研究与开发。
现代网页抓取面临动态、交互式网站的挑战，传统静态HTML解析方法常失效且需人工定制。为此，我们提出Webscraper框架，利用多模态大语言模型（MLLM）自主导航交互界面、调用专用工具并完成结构化数据提取，适用于传统爬虫无法应对的场景。该框架采用结构化的五阶段提示流程及自研工具，针对常见的“索引-内容”架构网站进行设计。在6个新闻网站上的实验表明，配备引导提示与专用工具的完整Webscraper框架，在数据提取准确率上显著优于基线智能体Anthropic's Computer Use。此外，该框架还成功应用于电商平台，验证了其泛化能力。
原文摘要 · Abstract (English)
Modern web scraping struggles with dynamic, interactive websites that require more than static HTML parsing. Current methods are often brittle and require manual customization for each site. To address this, we introduce Webscraper, a framework designed to handle the challenges of modern, dynamic web applications. It leverages a Multimodal Large Language Model (MLLM) to autonomously navigate interactive interfaces, invoke specialized tools, and perform structured data extraction in environments where traditional scrapers are ineffective. Webscraper utilizes a structured five-stage prompting procedure and a set of custom-built tools to navigate and extract data from websites following the common ``index-and-content'' architecture. Our experiments, conducted on six news websites, demonstrate that the full Webscraper framework, equipped with both our guiding prompt and specialized tools, achieves a significant improvement in extraction accuracy over the baseline agent Anthropic's Computer Use. We also applied the framework to e-commerce platforms to validate its generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。