对比多种嵌入模型与池化策略,提升电子病历信息检索效果。
Lessons Learned on Information Retrieval in Electronic Health Records: A Comparison of Embedding Models and Pooling Strategies
- 测试7种嵌入模型和池化方法在病历检索中的表现。
- 通用小模型BGE在多数场景下优于专业医疗模型。
- 模型选择、池化策略和查询表达对结果影响显著。
目标:将大语言模型应用于临床领域面临病历文本上下文密集的挑战。检索增强生成(RAG)通过在大规模文本源上进行推理提供解决方案,但仅检索系统就有众多参数需优化。本文开展消融实验,探究不同嵌入模型与池化策略对临床领域信息检索的影响。方法:在两个电子病历(EHR)数据源上的三个检索任务中,对比了七种模型,包括医学与通用领域模型、专用编码器嵌入模型以及现成的解码器LLM,并独立评估每种模型在查询和待检索文本上的嵌入池化策略。结果:嵌入模型的选择显著影响检索性能,其中相对较小的通用领域模型BGE持续优于所有其他模型,包括医学专用模型。然而,研究也发现不同数据集和查询表述间存在显著差异。我们还确定了每种模型的最佳池化方法,为未来检索系统设计提供指导。讨论:嵌入模型、池化策略及查询构造均显著影响检索性能,且这些模型在公开基准上的表现并不一定可迁移至新领域。此类实证研究对临床领域检索框架(如RAG)的稳健开发至关重要。
原文摘要 · Abstract (English)
Objective: Applying large language models (LLMs) to the clinical domain is challenging due to the context-heavy nature of processing medical records. Retrieval-augmented generation (RAG) offers a solution by facilitating reasoning over large text sources. However, there are many parameters to optimize in just the retrieval system alone. This paper presents an ablation study exploring how different embedding models and pooling methods affect information retrieval for the clinical domain. Methods: Evaluating on three retrieval tasks on two electronic health record (EHR) data sources, we compared seven models, including medical- and general-domain models, specialized encoder embedding models, and off-the-shelf decoder LLMs. We also examine the choice of embedding pooling strategy for each model, independently on the query and the text to retrieve. Results: We found that the choice of embedding model significantly impacts retrieval performance, with BGE, a comparatively small general-domain model, consistently outperforming all others, including medical-specific models. However, our findings also revealed substantial variability across datasets and query text phrasings. We also determined the best pooling methods for each of these models to guide future design of retrieval systems. Discussion: The choice of embedding model, pooling strategy, and query formulation can significantly impact retrieval performance and the performance of these models on other public benchmarks does not necessarily transfer to new domains. Further studies such as this one are vital for guiding empirically-grounded development of retrieval frameworks, such as in the context of RAG, for the clinical domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。