用可控数据集测试大模型从混乱网页中精准找证据的能力
Guardian Crawler: Retrieval-First Knowledge Discovery with Bounded LLM Augmentation for Noisy Web Intelligence
- 先检索后生成，结合关键词与语义增强重排
- 在900文档数据上达P@10=1.00，NDCG@10=0.94
- 适合做信息验证、自动摘要的算法评估
从噪声网页数据中提取相关证据极具挑战，尤其在包含不完整报告、语言混杂和无关内容的敏感领域。本文提出Guardian Crawler，一个可复现的检索优先测试平台，用于在合成网页类语料上进行知识发现与证据支撑摘要的受控实验。该架构结合BM25检索与风险感知、嵌入增强及混合重排，并采用受限的检索增强生成，显式引用文献来源。在900文档合成语料与10个查询的实验中，基于风险的重排达到最高描述性检索效果，P@10 = 1.00，NDCG@10 = 0.94，优于BM25的0.94和0.81。最佳混合与BM25+语义配置的NDCG@10分别为0.94和0.88。所有41个可评估生成条目均通过词汇覆盖率阈值；自动化大模型判别显示36条获支持，1条部分支持，4条不支持。结果表明Guardian Crawler具备作为受控测试平台的可行性，但未建立统计优势、人工验证的真实性或对真实网络调查环境的迁移能力。
原文摘要 · Abstract (English)
Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content. We present Guardian Crawler, a reproducible retrieval-first testbed for controlled experiments on knowledge discovery and evidence-grounded summarization over synthetic web-like corpora. The architecture combines BM25 retrieval with risk-aware, embedding-augmented, and hybrid reranking, followed by constrained retrieval-augmented generation with explicit document citations. Experiments on a synthetic 900-document corpus and 10 queries produced the highest descriptive retrieval scores under risk-based reranking, with P@10 = 1.00 and NDCG@10 = 0.94, compared with 0.94 and 0.81 for BM25. The best hybrid and BM25+Semantic configurations reached NDCG@10 values of 0.94 and 0.88, respectively. All 41 evaluable generated bullets passed the lexical coverage threshold; an automated LLM judge classified 36 as supported, one as partially supported, and four as unsupported. These results demonstrate the feasibility of Guardian Crawler as a controlled testbed but do not establish statistical superiority, human-validated faithfulness, or transfer to live-web investigative environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。