用大模型自动分类网页类型，提升爬虫发现新页面的效率
Web Page Classification using LLMs for Crawling Support
- 用大语言模型区分索引页与内容页，指导爬虫优先访问索引页
- 在自建数据集上分类准确率显著高于传统方法，覆盖更多新页面
- 适合需要高效抓取海量网页的网络爬虫系统开发者
网络爬虫需高效获取新网页，传统依赖站点地图和更新频率等特征的方法在不同场景下泛化能力有限。本文提出一种基于大语言模型（LLM）的网页分类方法，将网页分为“索引页”和“内容页”两类，并利用分类结果选择索引页作为新页面的入口点。研究构建了包含自动标注网页类型的语料库，从分类性能和新页面覆盖率两方面评估方法效果。实验表明，该方法在分类准确率和新页面覆盖范围上均优于基线模型，在真实爬虫任务中表现出更强的适应性与有效性。
原文摘要 · Abstract (English)
A web crawler is a system designed to collect web pages, and efficient crawling of new pages requires appropriate algorithms. While website features such as XML sitemaps and the frequency of past page updates provide important clues for accessing new pages, their universal application across diverse conditions is challenging. In this study, we propose a method to efficiently collect new pages by classifying web pages into two types, "Index Pages" and "Content Pages," using a large language model (LLM), and leveraging the classification results to select index pages as starting points for accessing new pages. We construct a dataset with automatically annotated web page types and evaluate our approach from two perspectives: the page type classification performance and coverage of new pages. Experimental results demonstrate that the LLM-based method outperformed baseline methods in both evaluation metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。