用强化学习生成可复用脚本，高效提取网页结构化数据
SCRIBES: Web-Scale Script-Based Semi-Structured Data Extraction with Reinforcement Learning
- 通过网页布局相似性设计奖励机制，生成跨页面通用提取脚本
- 在CommonCrawl数据上迭代训练，脚本质量提升超13%，问答准确率增4%以上
- 适合需要大规模、低成本提取网页信息的研究者与工程师
HTML表格、列表和信息框中的半结构化内容占网络事实数据的很大比例，但格式复杂，可靠提取仍具挑战。现有方法或泛化能力差，或因每页调用大模型而资源消耗高。本文提出SCRIBES（基于脚本的网页规模半结构化数据抽取），利用同一网站内页面间的布局相似性作为奖励信号，生成可复用于结构相似页面组的通用提取脚本。方法通过在真实世界CommonCrawl数据上生成合成标注进行迭代训练。实验表明，该方法在脚本质量上优于强基线超过13%，使GPT-4o下游问答准确率提升4%以上，实现可扩展且资源高效的网络信息抽取。
原文摘要 · Abstract (English)
Semi-structured content in HTML tables, lists, and infoboxes accounts for a substantial share of factual data on the web, yet the formatting complicates usage, and reliably extracting structured information from them remains challenging. Existing methods either lack generalization or are resource-intensive due to per-page LLM inference. In this paper, we introduce SCRIBES (SCRIpt-Based Semi-Structured Content Extraction at Web-Scale), a novel reinforcement learning framework that leverages layout similarity across webpages within the same site as a reward signal. Instead of processing each page individually, SCRIBES generates reusable extraction scripts that can be applied to groups of structurally similar webpages. Our approach further improves by iteratively training on synthetic annotations from in-the-wild CommonCrawl data. Experiments show that our approach outperforms strong baselines by over 13% in script quality and boosts downstream question answering accuracy by more than 4% for GPT-4o, enabling scalable and resource-efficient web information extraction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。