医学大模型对提示词敏感，复杂提示反而降低准确率。
When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models
- 测试多种提示策略，发现思维链反使准确率下降5.7%。
- 打乱选项顺序导致预测改变59.1%，准确率最高降27.4个百分点。
- 直接选概率最高选项比提示法更准，适合医疗领域应用。
大型语言模型在医疗场景中应用日益广泛，但其对提示格式的敏感性尚未明确。本文在MedMCQA（4,183题）和PubMedQA（1,000题）上评估了MedGemma（4B与27B参数）的鲁棒性。结果表明：思维链（CoT）提示使准确率下降5.7%；少样本示例导致性能下降11.9%，位置偏差从0.14升至0.47；打乱答案选项使模型重选59.1%的情况，准确率最高下降27.4个百分点；前截断上下文至50%时准确率低于无上下文基线，而后截断保留97%全上下文准确率。此外，闭合填空评分（选择最高对数概率选项）在4B和27B模型上分别达到51.8%和64.5%，超越所有提示策略，揭示模型实际掌握知识远超生成文本所显示；排列投票可恢复4个百分点准确率。结果表明，通用模型验证的提示工程不适用于医疗领域，且存在更可靠的替代方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed in medical settings, yet their sensitivity to prompt formatting remains poorly characterized. We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions) across a broad suite of robustness tests. Our experiments reveal several concerning findings. Chain-of-Thought (CoT) prompting decreases accuracy by 5.7% compared to direct answering. Few-shot examples degrade performance by 11.9% while increasing position bias from 0.14 to 0.47. Shuffling answer options causes the model to change predictions 59.1% of the time, with accuracy dropping up to 27.4 percentage points. Front-truncating context to 50% causes accuracy to plummet below the no-context baseline, yet back-truncation preserves 97% of full-context accuracy. We further show that cloze scoring (selecting the highest log-probability option token) achieves 51.8% (4B) and 64.5% (27B), surpassing all prompting strategies and revealing that models "know" more than their generated text shows. Permutation voting recovers 4 percentage points over single-ordering inference. These results demonstrate that prompt engineering techniques validated on general-purpose models do not transfer to domain-specific medical LLMs, and that reliable alternatives exist.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。