构建标准化评估框架,对比传统方法与大模型在网页数据提取中的表现。
NEXT-EVAL: Next Evaluation of Traditional and LLM Web Data Record Extraction
- 基于MHTML生成数据集,用XPath标注并设计结构感知评分机制。
- Flat JSON输入使大模型F1达0.9567,显著降低位置幻觉。
- 适合研究网页抽取、评估大模型性能的学者和开发者参考。
有效评估网页数据记录提取方法至关重要,但受限于静态、领域特定的基准和不透明的评分方式。这使得依赖结构启发式规则的传统算法与能跨多样布局实现零样本提取的大语言模型(LLM)之间的公平比较尤为困难。为此,我们提出一个具体评估框架:从任意MHTML快照系统生成评估数据集,标注基于XPath的监督标签,并采用结构感知指标进行一致评分,特别防止文本幻觉,仅评估位置幻觉。同时引入预处理策略优化输入以保留DOM语义:HTML瘦身、分层JSON和扁平化JSON。此外,通过转换DOM结构和修改内容创建公开可获取的合成数据集。我们在多个输入格式上对确定性启发式算法和现成的LLM进行基准测试。结果显示,扁平化JSON输入使LLM在提取准确率(F1分数0.9567)上表现最优,且幻觉最少。该框架建立了严谨评估的标准基础,为网页数据记录提取的下一步系统性进步铺平道路。
原文摘要 · Abstract (English)
Effective evaluation of web data record extraction methods is crucial, yet hampered by static, domain-specific benchmarks and opaque scoring practices. This makes fair comparison between traditional algorithmic techniques, which rely on structural heuristics, and Large Language Model (LLM)-based approaches, offering zero-shot extraction across diverse layouts, particularly challenging. To overcome these limitations, we introduce a concrete evaluation framework. Our framework systematically generates evaluation datasets from arbitrary MHTML snapshots, annotates XPath-based supervision labels, and employs structure-aware metrics for consistent scoring, specifically preventing text hallucination and allowing only for the assessment of positional hallucination. It also incorporates preprocessing strategies to optimize input for LLMs while preserving DOM semantics: HTML slimming, Hierarchical JSON, and Flat JSON. Additionally, we created a publicly available synthetic dataset by transforming DOM structures and modifying content. We benchmark deterministic heuristic algorithms and off-the-shelf LLMs across these multiple input formats. Our benchmarking shows that Flat JSON input enables LLMs to achieve superior extraction accuracy (F1 score of 0.9567) and minimal hallucination compared to other input formats like Slimmed HTML and Hierarchical JSON. We establish a standardized foundation for rigorous benchmarking, paving the way for the next principled advancements in web data record extraction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。