用稀疏自编码器让医学影像报告可解释,零微调也能达顶尖效果。
An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation
- 通过稀疏自编码器分解视觉模型特征,生成人类可懂的医学概念。
- 在MIMIC-CXR上指标媲美顶尖模型,训练资源减少90%以上。
- 适合需要透明决策过程的医疗AI场景,尤其关注可解释性研究者。
放射科服务需求空前增长,推动了自动化报告生成的研究。现有视觉-语言模型存在幻觉、缺乏可解释性且需昂贵微调。本文提出SAE-Rad,利用稀疏自编码器(SAEs)将预训练视觉变压器的潜在表示分解为人类可理解的特征。该混合架构结合前沿SAE技术,在保持稀疏性的同时实现高精度重构。使用现成语言模型,将真实报告蒸馏为每个SAE特征的描述,并组合生成完整报告,无需对大模型进行微调。据我们所知,SAE-Rad是首个将机制可解释性技术明确用于下游多模态推理任务的工作。在MIMIC-CXR数据集上,其辐射学特定指标与最先进模型相当,但训练计算资源显著降低。定性分析显示,SAE-Rad学习到有意义的视觉概念,生成报告与专家意见高度一致。结果表明,SAEs能增强医疗多模态推理,为现有VLM提供更可解释的替代方案。
原文摘要 · Abstract (English)
Radiological services are experiencing unprecedented demand, leading to increased interest in automating radiology report generation. Existing Vision-Language Models (VLMs) suffer from hallucinations, lack interpretability, and require expensive fine-tuning. We introduce SAE-Rad, which uses sparse autoencoders (SAEs) to decompose latent representations from a pre-trained vision transformer into human-interpretable features. Our hybrid architecture combines state-of-the-art SAE advancements, achieving accurate latent reconstructions while maintaining sparsity. Using an off-the-shelf language model, we distil ground-truth reports into radiological descriptions for each SAE feature, which we then compile into a full report for each image, eliminating the need for fine-tuning large models for this task. To the best of our knowledge, SAE-Rad represents the first instance of using mechanistic interpretability techniques explicitly for a downstream multi-modal reasoning task. On the MIMIC-CXR dataset, SAE-Rad achieves competitive radiology-specific metrics compared to state-of-the-art models while using significantly fewer computational resources for training. Qualitative analysis reveals that SAE-Rad learns meaningful visual concepts and generates reports aligning closely with expert interpretations. Our results suggest that SAEs can enhance multimodal reasoning in healthcare, providing a more interpretable alternative to existing VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。