用肝脏应激测试思路，发现医疗大模型隐藏安全缺陷。
Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy
- 设计叙事应激测试框架，模拟临床真实场景下的模型表现。
- 量化三指标：代谢指数、扰动翻转率、反事实公平性，揭示模型差异。
- 开源模型在安全维度优于闭源模型，提示需重视应激评估而非仅看准确率。
大型语言模型（LLMs）正基于基准准确率进入临床实践，但该指标可能无法检测出关键的安全失效模式。本文提出AI-MASLD应激审计框架，借鉴肝病学中的代谢应激测试逻辑，评估临床LLMs。通过六个叙事扰动探针的240个临床案例，对七种模型进行双重应激测试，并以代谢指数（MI）、扰动翻转率（PFR）和反事实公平性指数（CFI）三个指标量化性能。在清洁基线条件下，所有模型表现一致良好；但在真实叙事应激下，性能显著分化，暴露出两种截然不同的应激反应表型。量化模型表现出伪正常化现象，低翻转率掩盖功能崩溃；医学监督微调系统性降低逻辑稳定性、公平性和信息提取能力。一个开源模型在所有安全维度上达到或超过专有模型表现。研究确立叙事应激审计是准确性评估的必要补充。
原文摘要 · Abstract (English)
Large language models (LLMs) are entering clinical practice based on benchmark accuracy that may fail to detect safety-relevant failure modes. Here we present AI-MASLD, a stress-audit framework that adapts the logic of metabolic stress testing from hepatology to the evaluation of clinical LLMs. Using 240 clinical cases across six narrative perturbation probes, we subjected seven models to double-stress testing and quantified performance through three indices: metabolic index (MI), perturbation flip rate (PFR), and counterfactual fairness index (CFI). Under clean baseline conditions, all models performed uniformly well. Under realistic narrative stress, performance diverged sharply, revealing two distinct stress-response phenotypes. Quantized models exhibited pseudonormalization, in which low flip rates hid functional collapse. Medical supervised fine-tuning systematically degraded logical stability, fairness, and information extraction. An open-weight model matched or exceeded proprietary alternatives on every safety dimension. These findings establish narrative stress auditing as a necessary complement to accuracy-based evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。