小模型在医疗问答中表现不稳定，需同时评估准确性和回答一致性。
Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare
- 用五种不同提问方式测试五个开源小模型，均本地运行无微调。
- Llama 3.2 准确率最高（49.0%-65.0%），一致性适中；Gemma 2 一致性强但准确率低（33.0%-43.5%）。
- 角色扮演提示会显著降低准确率，且领域预训练模型易忽略指令，临床使用风险高。
小型开源语言模型在缺乏云基础设施和GPU的低资源医疗环境中日益受到关注。然而，这些模型在相同临床问题的不同表述下表现是否可靠仍不明确。本文在三个临床问答数据集（MedQA、MedMCQA、PubMedQA）上，对五个开源模型（Gemma 2 2B、Phi-3 Mini 3.8B、Llama 3.2 3B、Mistral 7B、Meditron-7B）采用五种提示风格（原始、正式、简化、角色扮演、直接）进行评估，所有实验在消费级CPU上本地完成，未进行微调。通过一致性分数、准确率和指令遵循失败率分析模型行为。结果显示，一致性与准确率在各模型间基本独立：Gemma 2 一致性最高（0.845-0.888），但准确率最低（33.0-43.5%）；Llama 3.2 准确率最高（49.0-65.0%），一致性中等（0.774-0.807）。角色扮演提示显著降低所有模型准确率，其中 Phi-3 Mini 在 MedQA 上下降 21.5 个百分点。Meditron-7B 在 PubMedQA 上指令遵循失败率达 99.0%，表明仅领域预训练不足以支撑结构化临床问答。研究揭示高一致性不代表正确性，模型可能稳定地给出错误答案，这对临床AI构成严重风险。综合来看，Llama 3.2 在准确率与可靠性之间表现最佳，适用于低资源部署。安全的临床AI需联合评估一致性、准确率与指令遵循能力。
原文摘要 · Abstract (English)
Small open-source language models are gaining attention for healthcare applications in low-resource settings where cloud infrastructure and GPU hardware may be unavailable. However, the reliability of these models under different phrasings of the same clinical question remains poorly understood. We evaluate five open-source models (Gemma 2 2B, Phi-3 Mini 3.8B, Llama 3.2 3B, Mistral 7B, and Meditron-7B, a domain-pretrained model without instruction tuning) across three clinical question answering datasets (MedQA, MedMCQA, and PubMedQA) using five prompt styles: original, formal, simplified, roleplay, and direct. Model behavior is evaluated using consistency scores, accuracy, and instruction-following failure rates. All experiments were conducted locally on consumer CPU hardware without fine-tuning. Consistency and accuracy were largely independent across models. Gemma 2 achieved the highest consistency (0.845-0.888) but the lowest accuracy (33.0-43.5%), while Llama 3.2 showed moderate consistency (0.774-0.807) alongside the highest accuracy (49.0-65.0%). Roleplay prompts consistently reduced accuracy across all models, with Phi-3 Mini dropping 21.5 percentage points on MedQA. Meditron-7B exhibited near-complete instruction-following failure on PubMedQA (99.0% UNKNOWN rate), indicating that domain pretraining alone is insufficient for structured clinical question answering. These findings show that high consistency does not imply correctness: models can be reliably wrong, a dangerous failure mode in clinical AI. Llama 3.2 demonstrated the strongest balance of accuracy and reliability for low-resource deployment. Safe clinical AI requires joint evaluation of consistency, accuracy, and instruction adherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。