用强化学习高效抓取网页中的统计资源，只爬少量页面就拿到多数数据。
Efficient Crawling for Scalable Web Data Acquisition (Extended Version)
- 基于睡眠老虎机的强化学习，智能筛选高价值链接路径。
- 在百万级网页网站上，仅爬少量页面就获取超九成目标数据。
- 适合需要大规模获取公开统计数据的研究者或数据工程师。
新闻事实核查及社会经济研究需分析高质量统计数据集（SD）。然而，由于数据发布方式不同，大规模获取这些数据集可能困难、低效甚至不可能。为提升开放统计数据的可获取性，我们提出一种聚焦式网络爬虫算法，能以高效且可扩展的方式，从给定网站中尽可能多地抓取特定类型的目标资源，而无需爬取整个网站。我们证明该问题最优求解是不可行的，并提出基于强化学习的方法——睡眠老虎机（sleeping bandits）。我们设计了SB-CLASSIFIER爬虫，通过分析链接所在页面的路径，学习哪些超链接更可能指向包含大量目标的页面。在包含数百万网页的网站上的实验表明，该爬虫极为高效，仅需爬取极小部分网页即可获得站点中绝大部分目标数据。
原文摘要 · Abstract (English)
Journalistic fact-checking, as well as social or economic research, require analyzing high-quality statistics datasets (SDs, in short). However, retrieving SD corpora at scale may be hard, inefficient, or impossible, depending on how they are published online. To improve open statistics data accessibility, we present a focused Web crawling algorithm that retrieves as many targets, i.e., resources of certain types, as possible, from a given website, in an efficient and scalable way, by crawling (much) less than the full website. We show that optimally solving this problem is intractable, and propose an approach based on reinforcement learning, namely using sleeping bandits. We propose SB-CLASSIFIER, a crawler that efficiently learns which hyperlinks lead to pages that link to many targets, based on the paths leading to the links in their enclosing webpages. Our experiments on websites with millions of webpages show that our crawler is highly efficient, delivering high fractions of a site's targets while crawling only a small part.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。