用索引预测取代逐字生成，快速精准提取网页关键内容
An Index-based Approach for Efficient and Effective Web Content Extraction
- 将网页分段并预测内容位置索引，避免逐词生成延迟
- 在主内容和查询相关提取任务中均实现更高准确率与速度
- 适合需要高效处理海量网页的智能代理与检索增强系统
随着网络代理（如 Deep Research）频繁访问大量网页以收集分析信息，大上下文预算下信号密度低的 LLM 上下文管理成为智能体与 RAG 流水线中的核心挑战。现有内容提取方法存在不足：生成式模型延迟高，规则启发式缺乏适应性，分块重排序方法无视网页结构。为此，我们提出基于索引的网页内容提取方法，将提取过程从慢速的逐令牌生成重构为高效的索引预测任务，兼顾效果与效率。通过结构感知、可定位的 HTML 分段，仅提取与查询相关的片段位置索引，使提取延迟与内容长度解耦，实现快速、精准的查询相关提取。我们在 RAG QA 系统中作为后检索处理组件评估，提升问答准确率；并在主内容提取（ME）与查询相关提取（QE）两种场景下直接测量目标内容匹配率。实验表明，该方法在准确率与速度上均优于现有工作，有效弥合了 LLM 与海量网页之间的差距。
原文摘要 · Abstract (English)
As web agents (e.g., Deep Research) routinely consume massive volumes of web pages to gather and analyze information, LLM context management -- under large token budgets and low signal density -- emerges as a foundational, high-importance, and technically challenging problem for agentic and RAG pipelines. Existing solutions for extracting relevant content are inadequate: generative extraction models suffer from high latency, rule-based heuristics lack adaptability, and chunk-and-rerank methods are blind to webpage structure. To overcome these issues, we introduce Index-based Web Content Extraction to reframe the extraction process from slow, token-by-token generation into a highly efficient, discriminative task of index prediction, achieving both effectiveness and efficiency. We partition HTML into structure-aware, addressable segments, and extract only the positional indices of content relevant to a given query. This method decouples extraction latency from content length, enabling rapid, query-relevant extraction. We first evaluate our method as a post-retrieval processing component within an RAG QA system and find that it improves QA accuracy. Then we directly measure its match rate with the target content in two scenarios: main content extraction (ME) and query-relevant extraction (QE). Experimental results show that our method outperforms existing works in both accuracy and speed, effectively bridging the gap between LLMs and the vast webpages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。