对比三种架构,发现高级检索增强生成能显著减少政策问答中的幻觉。
Chunking, Retrieval, and Re-ranking: An Empirical Evaluation of RAG Architectures for Policy Document Question Answering
- 采用分块+检索+重排序的三阶段框架提升答案可靠性。
- 高级RAG使事实正确率从0.347提升至0.797,大幅降低幻觉。
- 适合需要高信息准确性的公共政策、医疗监管等严谨场景使用。
大型语言模型(LLM)在公共卫生政策领域的应用,为处理美国疾控中心(CDC)等机构维护的海量法规文件提供了新途径。然而,LLM易产生看似合理却错误的幻觉,严重阻碍其在高风险环境中落地。本研究通过实证评估检索增强生成(RAG)架构在缓解该问题上的效果,比较了基础模型、基础RAG与带交叉编码器重排序的高级RAG方案。实验采用Mistral-7B-Instruct-v0.2模型和all-MiniLM-L6-v2嵌入模型,基于官方CDC政策分析框架与指导文档构建数据集。评估两种分块策略——递归字符级与基于词元的语义分割——对系统准确性的影响,以忠实度和相关性评分衡量。结果表明,基础RAG相比基线(0.347)将忠实度提升至0.621;而高级RAG进一步达到0.797的平均忠实度,验证了两阶段检索机制对领域特定政策问答的重要性。尽管如此,文档切分结构限制仍是多步推理任务的主要瓶颈。
原文摘要 · Abstract (English)
The integration of Large Language Models (LLMs) into the public health policy sector offers a transformative approach to navigating the vast repositories of regulatory guidance maintained by agencies such as the Centers for Disease Control and Prevention (CDC). However, the propensity for LLMs to generate hallucinations, defined as plausible but factually incorrect assertions, presents a critical barrier to the adoption of these technologies in high-stakes environments where information integrity is non-negotiable. This empirical evaluation explores the effectiveness of Retrieval-Augmented Generation (RAG) architectures in mitigating these risks by grounding generative outputs in authoritative document context. Specifically, this study compares a baseline Vanilla LLM against Basic RAG and Advanced RAG pipelines utilizing cross-encoder re-ranking. The experimental framework employs a Mistral-7B-Instruct-v0.2 model and an all-MiniLM-L6-v2 embedding model to process a corpus of official CDC policy analytical frameworks and guidance documents. The analysis measures the impact of two distinct chunking strategies, recursive character-based and token-based semantic splitting, on system accuracy, measured through faithfulness and relevance scores across a curated set of complex policy scenarios. Quantitative findings indicate that while Basic RAG architectures provide a substantial improvement in faithfulness (0.621) over Vanilla baselines (0.347), the Advanced RAG configuration achieves a superior faithfulness average of 0.797. These results demonstrate that two-stage retrieval mechanisms are essential for achieving the precision required for domain-specific policy question answering, though structural constraints in document segmentation remain a significant bottleneck for multi-step reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。