用LLM训练偏好优化网页爬取，21%数据量达同等效果
Craw4LLM: Efficient Web Crawling for LLM Pretraining
- 基于LLM预训练影响力动态调度爬取优先级
- 仅爬21%网页即达同等下游性能
- 适合大规模预训练数据构建者使用
网页爬取是大语言模型预训练数据的主要来源，但多数爬取页面因质量低被丢弃。本文提出Craw4LLM，一种基于LLM预训练偏好探索网页图的高效爬取方法。它将网页在预训练中的影响作为爬虫调度优先级，取代传统的基于图连通性的优先级。在包含9亿网页的商业搜索引擎索引图上实验表明，仅需爬取21%的URL，Craw4LLM预训练的模型即可达到先前方法的下游性能，显著减少爬取浪费并减轻对网站的压力。代码已公开于https://github.com/cxcscmu/Craw4LLM。
原文摘要 · Abstract (English)
Web crawl is a main source of large language models' (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper presents Craw4LLM, an efficient web crawling method that explores the web graph based on the preference of LLM pretraining. Specifically, it leverages the influence of a webpage in LLM pretraining as the priority score of the web crawler's scheduler, replacing the standard graph connectivity based priority. Our experiments on a web graph containing 900 million webpages from a commercial search engine's index demonstrate the efficiency of Craw4LLM in obtaining high-quality pretraining data. With just 21% URLs crawled, LLMs pretrained on Craw4LLM data reach the same downstream performances of previous crawls, significantly reducing the crawling waste and alleviating the burdens on websites. Our code is publicly available at https://github.com/cxcscmu/Craw4LLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。