评测大模型在真实医疗数据标准下的问答能力，推动临床AI落地
FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering
- 基于真实临床问题和FHIR数据标准构建评测基准
- 发现直接调用API与代码推理更有效，多轮交互提升准确性
- 适合医疗AI研究者和临床系统开发者参考
随着健康信息交换标准HL7 FHIR的普及，临床人工智能面临新挑战：大模型需在复杂的资源型数据模型中完成任务，而非传统结构化数据。然而现有评测基准滞后于这一转变，缺乏对真实场景的覆盖。为此，我们提出FHIR-AgentBench，将2,931个真实临床问题锚定于FHIR标准，系统评估不同代理框架的表现。实验比较了直接调用FHIR API与专用工具、单轮与多轮交互、自然语言与代码生成等策略。结果表明，从复杂FHIR资源中获取数据及推理过程均具挑战性，显著影响问答性能。我们公开发布FHIR-AgentBench数据集与评估工具（https://github.com/glee4810/FHIR-AgentBench），以支持可复现研究和可靠临床大模型代理的开发。
原文摘要 · Abstract (English)
The recent shift toward the Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR) standard opens a new frontier for clinical AI, demanding LLM agents to navigate complex, resource-based data models instead of conventional structured health data. However, existing benchmarks have lagged behind this transition, lacking the realism needed to evaluate recent LLMs on interoperable clinical data. To bridge this gap, we introduce FHIR-AgentBench, a benchmark that grounds 2,931 real-world clinical questions in the HL7 FHIR standard. Using this benchmark, we systematically evaluate agentic frameworks, comparing different data retrieval strategies (direct FHIR API calls vs. specialized tools), interaction patterns (single-turn vs. multi-turn), and reasoning strategies (natural language vs. code generation). Our experiments highlight the practical challenges of retrieving data from intricate FHIR resources and the difficulty of reasoning over them, both of which critically affect question answering performance. We publicly release the FHIR-AgentBench dataset and evaluation suite (https://github.com/glee4810/FHIR-AgentBench) to promote reproducible research and the development of robust, reliable LLM agents for clinical applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。