3D脑影像通用模型,无需重训即可适应多种任务。
Neuroverse3D: Developing In-Context Learning Universal Model for Neuroimaging in 3D
- 用自适应并串处理和融合策略降低3D输入内存开销。
- 在14项任务上超越现有模型,接近专用模型性能。
- 适合医疗中心间差异大、需快速适配的场景。
上下文学习(ICL)是一种无需重训练即可跨任务泛化的通用模型,通过上下文中的任务引导实现高效适应,尤其适合复杂的脑影像分析需求。然而,现有ICL模型受限于2D输入,难以扩展到3D数据,因高内存消耗导致性能下降。为此,我们提出Neuroverse3D,一个可在3D脑影像中执行多任务(如分割、去噪、修复)的ICL模型。该模型通过自适应并串式上下文处理与U型融合策略,有效缓解3D输入带来的内存压力,支持无限数量的上下文图像。同时,设计优化损失函数以平衡多任务训练并强化解剖边界关注。研究整合了来自19个脑影像数据集的43,674例3D多模态扫描,在14项不同任务上使用预留测试集进行评估。结果表明,Neuroverse3D显著优于现有ICL模型,且性能接近专用模型,可在不重训的前提下灵活适应不同医疗中心的数据差异。代码与模型权重已公开于https://github.com/jiesihu/Neuroverse3D。
原文摘要 · Abstract (English)
In-context learning (ICL), a type of universal model, demonstrates exceptional generalization across a wide range of tasks without retraining by leveraging task-specific guidance from context, making it particularly effective for the intricate demands of neuroimaging. However, current ICL models, limited to 2D inputs and thus exhibiting suboptimal performance, struggle to extend to 3D inputs due to the high memory demands of ICL. In this regard, we introduce Neuroverse3D, an ICL model capable of performing multiple neuroimaging tasks in 3D (e.g., segmentation, denoising, inpainting). Neuroverse3D overcomes the large memory consumption associated with 3D inputs through adaptive parallel-sequential context processing and a U-shaped fusion strategy, allowing it to handle an unlimited number of context images. Additionally, we propose an optimized loss function to balance multi-task training and enhance focus on anatomical boundaries. Our study incorporates 43,674 3D multi-modal scans from 19 neuroimaging datasets and evaluates Neuroverse3D on 14 diverse tasks using held-out test sets. The results demonstrate that Neuroverse3D significantly outperforms existing ICL models and closely matches task-specific models, enabling flexible adaptation to medical center variations without retraining. The code and model weights are publicly available at https://github.com/jiesihu/Neuroverse3D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。