用稀疏自编码器解析医疗序列模型，发现特征抽象随深度递进，可提升死亡预测效果。
Sparse Autoencoder Decomposition of Clinical Sequence Model Representations: Feature Complexity, Task Specialisation, and Mortality Prediction
- 在10个残差层上训练稀疏自编码器，逐层分析临床序列特征
- 高层特征覆盖30种临床类型，对死亡预测优于密集表示
- 方法适合研究医疗模型内部机制，但特征稳定性较低
稀疏自编码器（SAEs）已用于大语言模型和蛋白质语言模型，但尚未系统应用于电子健康记录（EHR）基础模型。我们在一个1450万参数的自回归临床序列模型FlatASCEND上，于INSPECT（门诊）和MIMIC-IV（ICU）数据集的所有10个残差流提取点训练TopK SAE。SAE分解揭示了变压器深度上的渐进抽象：第0层特征接近完美标记检测器（45.7%单例），而第6层特征涵盖约30种不同临床类别的标记（0.5%单例）。在全序列简单线性探测中，SAE特征在离散事件预测（死亡率）上优于密集表示，但在连续量预测（住院时长）上表现较差——这一探测级别表征现象在临床相关的泄漏安全窗口中不成立：密集表示在所有测试设置中与或超过SAE特征（eICU-CRD 48小时AUC：SAE 0.871 vs 密集 0.880；基线模型零样本，SAE字典基于eICU激活训练；MIMIC-IV：0.836 vs 0.914；INSPECT 1年/3年：0.697 vs 0.800）。一种增量模式干预方法将SAE扰动噪声降低86倍，实现更清晰的特征级实验，尽管扰动效应在4种条件中的3种大于随机控制，但未达统计显著性。特征在不同随机种子间的可重现性仅为21%，个体特征应视为示例而非稳定实体。
原文摘要 · Abstract (English)
Sparse autoencoders (SAEs) have been applied to large language models and protein language models, but not systematically to electronic health record (EHR) foundation models. We train TopK SAEs on FlatASCEND, a 14.5-million-parameter autoregressive clinical sequence model, at all 10 residual stream extraction points on INSPECT (outpatient) and MIMIC-IV (ICU). SAE decomposition reveals progressive abstraction across transformer depth: layer-0 features are near-perfect token detectors (45.7% singleton), while layer-6 features span approximately 30 token types across multiple clinical categories (0.5% singleton). Under full-sequence simple linear probes, SAE features outperform dense representations for discrete event prediction (mortality) while dense representations outperform for continuous magnitude prediction (length of stay) - a probe-level representational phenomenon that does not extend to clinically relevant leakage-safe windows, where dense representations match or exceed SAE features across all tested settings (eICU-CRD 48-hour AUC: SAE 0.871 versus dense 0.880; base model zero-shot, SAE dictionaries trained on eICU activations; MIMIC-IV: 0.836 versus 0.914; INSPECT 1-year/3-year: 0.697 versus 0.800). A delta-mode intervention method reduces SAE perturbation noise by 86x, enabling cleaner feature-level experiments, though the resulting perturbation effects are larger than random controls in 3 of 4 conditions but not formally significant. Feature reproducibility across random seeds is 21%, and individual features should be interpreted as illustrative rather than stable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。