用AI+提示词自动抓取网页，效果比传统方法更好
The Synergy of Automated Pipelines with Prompt Engineering and Generative AI in Web Crawling
- 用提示词引导AI生成抓取代码，分通用和特定元素两种策略
- Claude AI生成的代码在功能、可读性上优于ChatGPT-4.0
- 适合想自动化数据采集的研究者和开发者
网页抓取是获取在线数据的关键技术，但受限于页面多样性与反爬机制。本研究探索将生成式AI工具Claude AI（Sonnet 3.5）与ChatGPT-4.0结合提示词工程，实现网页抓取自动化。采用两个提示：PROMPT I（通用推理，测试于Yahoo News）和PROMPT II（元素特定，测试于Coupons.com），评估生成脚本的质量与性能。通过预设指标（功能性、可读性、模块化、鲁棒性）及三位评估员的结构化打分，结果表明Claude AI在脚本质量与适应性上持续优于ChatGPT-4.0。实验还集成undetected_chromedriver、Selenium与fake_useragent等反爬方案提升稳定性。可视化分析进一步验证了其优势。研究表明，生成式AI结合提示词工程能显著简化并优化网页抓取流程。
原文摘要 · Abstract (English)
Web crawling is a critical technique for extracting online data, yet it poses challenges due to webpage diversity and anti-scraping mechanisms. This study investigates the integration of generative AI tools Claude AI (Sonnet 3.5) and ChatGPT4.0 with prompt engineering to automate web scraping. Using two prompts, PROMPT I (general inference, tested on Yahoo News) and PROMPT II (element-specific, tested on Coupons.com), we evaluate the code quality and performance of AI-generated scripts. Claude AI consistently outperformed ChatGPT-4.0 in script quality and adaptability, as confirmed by predefined evaluation metrics, including functionality, readability, modularity, and robustness. Performance data were collected through manual testing and structured scoring by three evaluators. Visualizations further illustrate Claude AI's superiority. Anti-scraping solutions, including undetected_chromedriver, Selenium, and fake_useragent, were incorporated to enhance performance. This paper demonstrates how generative AI combined with prompt engineering can simplify and improve web scraping workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。