用神经网络评估网页质量，提升搜索系统爬取效率与结果相关性。
Document Quality Scoring for Web Crawling
- 基于神经网络的语义质量评分模型，用于网页优先级排序。
- 高质网页优先爬取可显著提升下游搜索效果。
- 提供开源Docker工具包，便于集成到现有搜索系统中。
互联网充斥着大量低质量内容，但用户期望搜索引擎返回高质量、相关性强的结果。低质量页面会浪费爬取和检索资源，影响系统性能。因此，高效的网页质量评估技术对搜索引擎至关重要。本文在Chang等人（2024）提出的神经语义质量估计算法基础上，将其扩展应用于网页爬取优先级判断。实验表明，优先爬取语义质量高的网页能有效提升后续搜索的准确性。我们还发布了包含质量评分功能的Docker容器，方便其他搜索系统模块直接集成使用。
原文摘要 · Abstract (English)
The internet contains large amounts of low-quality content, yet users expect web search engines to deliver high-quality, relevant results. The abundant presence of low-quality pages can negatively impact retrieval and crawling processes by wasting resources on these documents. Therefore, search engines can greatly benefit from techniques that leverage efficient quality estimation methods to mitigate these negative impacts. Quality scoring methods for web pages are useful for many processes typical for web search systems, including static index pruning, index tiering, and crawling. Building on work by Chang et al.~\cite{chang2024neural}, who proposed using neural estimators of semantic quality for static index pruning, we extend their approach and apply their neural quality scorers to assess the semantic quality of web pages in crawling prioritisation tasks. In our experimental analysis, we found that prioritising semantically high-quality pages over low-quality ones can improve downstream search effectiveness. Our software contribution consists of a Docker container that computes an effective quality score for a given web page, allowing the quality scorer to be easily included and used in other components of web search systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。