解决向量检索中语义纠缠问题,提升RAG系统精准度
Semantic Entanglement in Vector-Based Retrieval: A Formal Framework and Context-Conditioned Disentanglement Pipeline for Agentic RAG Systems
- 提出语义解缠管道SDP,重构文档结构以减少向量空间重叠
- 在医疗知识库上,检索精度从32%提升至82%,纠缠指数降至0.14
- 适合关注RAG系统优化与文档预处理的研究者和工程师
检索增强生成(RAG)系统依赖向量表示的几何特性来获取相关证据。当源文档在连续文本中混合多个主题时,标准向量化会使语义不同的内容占据重叠的向量邻域,这种现象称为语义纠缠。本文将纠缠定义为模型相关的跨主题重叠度量,并引入纠缠指数(EI)作为量化代理。高EI会限制余弦相似度检索下的Top-K精度。为此,提出语义解缠管道(SDP),一个四阶段预处理框架,在嵌入前重构文档。进一步提出基于使用模式的上下文感知预处理,以及根据智能体表现持续反馈调整文档结构。在包含2000多篇文档、约25个子领域的实际企业医疗知识库上评估,固定分块下的检索精度约为32%,而采用SDP后提升至约82%,平均EI从0.71降至0.14。虽不认为纠缠完全解释RAG失败,但其揭示了一种下游优化难以修正的预处理缺陷。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems depend on the geometric properties of vector representations to retrieve contextually appropriate evidence. When source documents interleave multiple topics within contiguous text, standard vectorization produces embedding spaces in which semantically distinct content occupies overlapping neighborhoods. We term this condition semantic entanglement. We formalize entanglement as a model-relative measure of cross-topic overlap in embedding space and define an Entanglement Index (EI) as a quantitative proxy. We argue that higher EI constrains attainable Top-K retrieval precision under cosine similarity retrieval. To address this, we introduce the Semantic Disentanglement Pipeline (SDP), a four-stage preprocessing framework that restructures documents prior to embedding. We further propose context-conditioned preprocessing, in which document structure is shaped by patterns of operational use, and a continuous feedback mechanism that adapts document structure based on agent performance. We evaluate SDP on a real-world enterprise healthcare knowledge base comprising over 2,000 documents across approximately 25 sub-domains. Top-K retrieval precision improves from approximately 32% under fixed-token chunking to approximately 82% under SDP, while mean EI decreases from 0.71 to 0.14. We do not claim that entanglement fully explains RAG failure, but that it captures a distinct preprocessing failure mode that downstream optimization cannot reliably correct once encoded into the vector space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。