解决文档分块检索重复问题,提升生成质量
Self-Conditioned Positional HNSW for Overlap-Aware Retrieval in Chunked-Document RAG Systems: Method and Industrial Evidence-Quality Audit
- 在嵌入向量中加入位置编码,用双阶段查询优化结果分布
- 实测770条评审中574条获3分以上,95%清晰截图可正确识别
- 适合需要高质量证据生成的工业级RAG系统
文档分块检索是检索增强生成(RAG)系统的核心组件。文档被分割为重叠块,经嵌入后通过近似最近邻搜索(如HNSW图)索引。重叠虽能提升边界覆盖,但导致前k项召回常返回相邻重复块,浪费提示预算。本文提出自条件位置HNSW(SCP-HNSW),通过在块嵌入中添加低维位置编码,并采用两阶段查询过程估计并应用查询相关的文档位置先验,实现无需修改原图结构的轻量改进。最终通过可审计的最小索引间距选择器构造上下文。同时整合工业级评审数据:770条文本证据审核中318条完全标注,70个案例的OCR审核含350个评分。文本审核显示770条中574条获3/5分,仅39条在1-2分区间,叙事细节远多于结构化问题标记;OCR审核显示清晰聊天截图通过率达95%,手写/模糊图像仅45%,评分一致性中到强。结果支持具备重叠感知与审计友好特性的RAG检索,并指出未来因果性能验证所需的关键控制性消融实验。
原文摘要 · Abstract (English)
Chunked-document retrieval is a common component of retrieval-augmented generation (RAG) systems. Documents are split into overlapping chunks, embedded, and indexed with approximate nearest-neighbor search such as hierarchical navigable small world graphs (HNSW). Overlap improves boundary coverage but induces a practical failure mode: top-k retrieval often returns near-adjacent chunks that repeat evidence and waste prompt budget. We propose Self-Conditioned Positional HNSW (SCP-HNSW), a lightweight modification that appends a low-dimensional positional code to chunk embeddings and uses a two-pass query procedure to estimate and apply a query-specific document-position prior. SCP-HNSW leaves HNSW graph construction and traversal unchanged while adding an auditable minimum-index-gap selector for final context construction. We also integrate industrial review artifacts for generated evidence quality: a 770-review text-evidence audit with 318 fully labeled reviews and a 70-case OCR audit with 350 ratings. The text audit shows that 574 of 770 projected reviews are rated 3/5, only 39 fall in the 1-2 range, and narrative reviewer detail appears much more often than structured issue flags. The OCR audit shows slice-level pass rates from 95% for clean chat screenshots to 45% for handwritten/blurry captures, with moderate to strong agreement. These results motivate overlap-aware, audit-friendly RAG retrieval and identify the remaining controlled retrieval ablations needed for causal performance claims.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。