评估小模型医疗问答时，发现输出一致性差，单次测试难反映真实可靠性。
Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework
- 构建开源评估框架，同时衡量准确率与多次生成结果的一致性。
- 三模型在50题上共1500次回答，87%-97%内容不同，一致性最高仅0.20。
- 提醒临床应用者：小模型未必更安全，需警惕领域微调与规模混淆。
将大语言模型用于医疗问答不仅需要高平均准确率，还需输出稳定可靠。在线健康社区如Reddit是数百万用户获取医疗信息的主要来源，但易受错误信息影响，部署LLM作为助手更需兼顾正确性与一致性。本文提出一个实用的开源评估框架，用于评测可本地部署的小型开放权重模型在医疗问答中的表现，将可复现性作为核心指标之一，与词汇和语义准确性并列。该流程计算八项质量指标（包括BERTScore、ROUGE-L及大模型判分），以及两项基于重复推理（每题10次）的内部一致性指标。在50个MedQuAD问题上对三个模型（Llama 3.1 8B、Gemma 3 12B、MedGemma 1.5 4B）进行评估，共生成1500条回答，结果显示即使使用低温生成（T=0.2），各模型自一致性最高仅为0.20，且87%-97%的回答彼此不同——现有单次评估完全无法捕捉这一安全缺口。临床微调的MedGemma 1.5 4B在质量和一致性上均低于更大的通用模型，但因其体积最小，此对比混杂了领域微调与模型规模的影响。方法细节详尽，可供从业者复制或扩展以支持自身模型选型流程。所有代码与数据管道已公开于 https://github.com/aviad-buskila/llm_medical_reproducibility。
原文摘要 · Abstract (English)
Incorporating large language models (LLMs) in medical question answering demands more than high average accuracy: a model that returns substantively different answers each time it is queried is not a reliable medical tool. Online health communities such as Reddit have become a primary source of medical information for millions of users, yet they remain highly susceptible to misinformation; deploying LLMs as assistants in these settings amplifies the need for output consistency alongside correctness. We present a practical, open-source evaluation framework for assessing small, locally-deployable open-weight LLMs on medical question answering, treating reproducibility as a first-class metric alongside lexical and semantic accuracy. Our pipeline computes eight quality metrics, including BERTScore, ROUGE-L, and an LLM-as-judge rubric, together with two within-model reproducibility metrics derived from repeated inference (N=10 runs per question). Evaluating three models (Llama 3.1 8B, Gemma 3 12B, MedGemma 1.5 4B) on 50 MedQuAD questions (N=1,500 total responses) reveals that despite low-temperature generation (T=0.2), self-agreement across runs reaches at most 0.20, while 87-97% of all outputs per model are unique -- a safety gap that single-pass benchmarks entirely miss. The clinically fine-tuned MedGemma 1.5 4B underperforms the larger general-purpose models on both quality and reproducibility; however, because MedGemma is also the smallest model, this comparison confounds domain fine-tuning with model scale. We describe the methodology in sufficient detail for practitioners to replicate or extend the evaluation for their own model-selection workflows. All code and data pipelines are available at https://github.com/aviad-buskila/llm_medical_reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。