arXiv:2608.01462cs.AI2026-08

提出可复用的多模态医疗模型失效分析框架,区分失败是响亮还是沉默。

Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI

  • 基于部署可观测信号,构建无需训练的模态失效分析框架。
  • 在真实数据中发现超声缺失使误差几乎翻倍,且多数失败无声无息。
  • 适用于临床模型上线前的风险评估,尤其关注模态缺失场景。

多模态临床模型通常在所有模态齐全时评估准确率,但实际部署时常缺失某些模态(如超声未普及而心电图常见)。此时不仅需关注准确率下降幅度,更需明确:哪个模态导致了失败?是明显异常(响亮)还是隐蔽错误(沉默)?本研究提出一种模型无关的模态失效分析框架,仅依赖部署阶段可观察的信号,即可对每个样本进行失效分类,生成模态间互补性矩阵,并识别出可监控的响亮失败与难以察觉的沉默失败。框架以小型、单元测试通过的工具形式发布,经人工构造的基准验证,能准确恢复预设的模态主导性和互补子集,且支持三模态分析。进一步应用于冻结的EchoJEPA和HuBERT-ECG嵌入,在配对的MIMIC-IV队列上对左室射血分数(LVEF)及射血分数≤40%的心衰(HFrEF)判断任务进行测试(n=245),结果显示移除超声后误差近翻倍;而超声与心电图重叠范围小这一发现本身即为心脏基础模型部署的重要启示。全部代码与结果见https://github.com/criticaldata/PRIMED-AI。

原文摘要 · Abstract (English)

Multimodal clinical models are usually judged on accuracy with every modality present, but deployment removes modalities; an echocardiogram is often unavailable where an ECG is routine. Two questions then matter beyond the size of the accuracy loss: which modality was responsible, and whether the model fails loudly or silently once that modality is dropped. The distinction is per-example and modality-level, and is separate from post-hoc feature attribution (e.g. SHAP). Models are replaced often; the evaluation that answers these questions is reused. We present a model-agnostic modality-failure framework: given N modality embeddings, any mask-aware probe, and labels, it returns a per-example failure taxonomy, a per-modality complementarity matrix that attributes error to modalities, and a loud-vs-silent dropout profile separating monitorable failures from those that pass unflagged far from the decision boundary, using only deployment-observable signals. We release it as a small, unit-tested harness and validate it against planted ground truth. Across seeds it recovers that planted modality dominance and complementary subset, reports per-modality loud-vs-silent rates, and scales to a three-modality complementarity matrix; because the planted structure is known by construction, this validates recovery of per-example attribution rather than clinical performance. We then instantiate the framework on frozen EchoJEPA and HuBERT-ECG embeddings for LVEF and the EF <= 40% HFrEF gate over a paired MIMIC-IV cohort, where on the held-out test split (n = 245) dropping echo nearly doubles error. The narrow echo-to-ECG overlap that bounds cohort size is itself a deployment finding for cardiac foundation models. All of our work can be found at https://github.com/criticaldata/PRIMED-AI.

多模态医疗AI失效分析模型可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。