用医学伦理哲学框架评估大模型临床决策的道德一致性
PrinciplismQA: A Philosophy-Grounded Approach to Assessing LLM-Human Clinical Medical Ethics Alignment
- 基于医学伦理原则构建系统化评估方法
- 3648道专家验证题揭示模型伦理短板
- 适合医疗AI安全评估与临床部署审查
随着医疗大模型迈向临床应用，评估其伦理推理能力变得至关重要。尽管在知识基准上表现优异，现有大模型缺乏对临床决策中多重合理方案间伦理权衡的有效评估工具。现有基准未系统融入公认哲学框架和专家验证。本文提出PrinciplismQA，一种基于医学伦理原则（Principlism）的评估方法，涵盖3648道由专家验证的题目，覆盖知识与临床推理。通过专家校准的评估流程，实现可复现评估并识别模型伦理偏差。实验发现，即使知识准确率高，多数模型仍存在显著伦理推理缺陷，表明知识训练无法保证临床伦理对齐。该工具可用于评估医疗AI部署准备度。数据与测试脚本已开源。
原文摘要 · Abstract (English)
As medical LLMs transition to clinical deployment, assessing their ethical reasoning capability becomes critical. While achieving high accuracy on knowledge benchmarks, LLMs lack validated assessment for navigating ethical trade-offs in clinical decision-making where multiple valid solutions exist. Existing benchmarks lack systematic approaches to incorporate recognized philosophical frameworks and expert validation for ethical reasoning assessment. We introduce PrinciplismQA, a philosophy-grounded approach to assessing LLM clinical medical ethics alignment. Grounded in Principlism, our approach provides a systematic methodology for incorporating clinical ethics philosophy into LLM assessment design. PrinciplismQA comprises 3,648 expert-validated questions spanning knowledge assessment and clinical reasoning. Our expert-calibrated pipeline enables reproducible evaluation and models ethical biases. Evaluating recent models reveals significant ethical reasoning gaps despite high knowledge accuracy, demonstrating that knowledge-oriented training does not ensure clinical ethical alignment. PrinciplismQA provides a validated tool for assessing clinical AI deployment readiness. Our data and test scripts are fully released on https://github.com/FreedomIntelligence/PrinciplismQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。