用检索增强生成提升公共健康问答可靠性,小模型也能超大模型。
Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

- 混合检索结合稠密与稀疏方法,提升召回率和排序质量。
- 引入上下文后,多选准确率显著提升,小模型可超越大模型。
- 设计评分体系评估自由回答,强调事实一致性的重要性。
大型语言模型在医学问答任务中表现良好,但在公共健康领域受限于幻觉问题及官方指引快速更新。检索增强生成(RAG)通过依赖显式维护的语料库来降低风险,但端到端性能高度依赖检索配置和评估方式。我们扩展了PubHealthBench基准,包含7,929个源自英国政府公共健康指南的问题,构建检索增强场景,并系统评估不同检索与生成策略。对比了稠密、稀疏与混合检索在多个嵌入模型和语料变体下的表现,发现混合检索在召回率和排序质量上持续领先,且块长度与主题交互影响排名效果。提供检索上下文后,各类LLM的多选准确率显著提升,使小型开源模型在无检索时的表现达到甚至超越大型模型。主要增益来自检索质量与上下文选择。为评估真实自由回答,我们引入基于评分标准的LLM作为裁判,涵盖忠实性、完整性、清晰度与事实一致性,并经双人人工标注验证。裁判与人类一致度最高的是忠实性和完整性,而事实一致性和清晰度较难复现,提示需谨慎解读这些维度的规模化结果。总体表明,检索是实现可靠公共健康问答的关键,为构建和评估基于官方指南的RAG系统提供了实用指导。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve promising results on medical question answering benchmarks, yet their use in public health is constrained by hallucinations and the rapid evolution of official guidance. Retrieval-Augmented Generation (RAG) mitigates these risks by grounding responses in an explicitly maintained corpus, but end-to-end performance depends critically on retrieval configuration and on evaluation beyond multiple-choice formats. We extend PubHealthBench, a question answering (QA) benchmark of 7,929 questions derived from UK Government public health guidance, into a retrieval-augmented setting and systematically evaluate retrieval and generation choices. We compare dense, sparse, and hybrid retrieval across multiple embedding models and corpus variants, and show that hybrid retrieval consistently improves recall and ranking quality, with chunk length and topic interacting with ranking performance. Providing retrieved context substantially increases multiple-choice accuracy across a diverse set of LLMs, enabling smaller open-weight models to match or outperform larger models used without retrieval, with gains primarily driven by retrieval quality and careful context selection. To assess realistic free-form answering, we introduce a rubric-based LLM-as-a-judge covering faithfulness, completeness, clarity, and factual consistency, and validate it against dual human annotations. Judge-human agreement is strongest for faithfulness and completeness, while factual consistency and clarity are less reliably reproduced, motivating caution when interpreting those dimensions at scale. Overall, our results highlight retrieval as a primary lever for reliable public health QA and provide practical guidance for building and evaluating RAG systems grounded in official guidance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。