arXiv:2602.08505cs.CV2026-02

视觉大模型在电子显微镜图像分割中表现不一,跨数据集泛化能力有限。

Are Vision Foundation Models Foundational for Electron Microscopy Image Segmentation?

  • 用轻量级头或LoRA微调视觉大模型进行分割
  • 单数据集训练效果好,多数据集联合训练反而性能下降
  • 不同电镜数据间存在深层特征差异,现有微调方法无法解决

尽管视觉基础模型(VFMs)在生物医学图像分析中越来越被复用,但其隐含表示是否具备足够的通用性以支持跨异构显微图像数据集的有效迁移仍不清楚。本文研究了电子显微镜(EM)图像中线粒体分割问题,采用两个公开的EM数据集(Lucchi++ 和 VNC)及三种代表性VFMs(DINOv2、DINOv3、OpenCLIP)。评估了两种实用的模型适配策略:冻结主干仅训练轻量分割头,以及通过低秩适应(LoRA)实现参数高效微调(PEFT)。结果显示,在所有主干模型上,单个EM数据集训练可获得良好分割性能(以前景交并比衡量),且LoRA能持续提升域内表现;而多个EM数据集联合训练导致所有模型性能严重下降,且PEFT仅带来微小改进。通过主成分分析、Fréchet DINOv2距离和线性探测等方法对潜在表示空间的探索表明,尽管两数据集视觉相似,但仍存在显著且持久的域间差异,这与联合训练失败现象一致。结果表明,虽然在单一域内轻量适配下视觉大模型可实现竞争力表现,但当前的参数高效微调策略不足以在无额外域对齐机制的情况下构建跨异构EM数据集的鲁棒统一模型。

原文摘要 · Abstract (English)

Although vision foundation models (VFMs) are increasingly reused for biomedical image analysis, it remains unclear whether the latent representations they provide are general enough to support effective transfer and reuse across heterogeneous microscopy image datasets. Here, we study this question for the problem of mitochondria segmentation in electron microscopy (EM) images, using two popular public EM datasets (Lucchi++ and VNC) and three recent representative VFMs (DINOv2, DINOv3, and OpenCLIP). We evaluate two practical model adaptation regimes: a frozen-backbone setting in which only a lightweight segmentation head is trained on top of the VFM, and parameter-efficient fine-tuning (PEFT) via Low-Rank Adaptation (LoRA) in which the VFM is fine-tuned in a targeted manner to a specific dataset. Across all backbones, we observe that training on a single EM dataset yields good segmentation performance (quantified as foreground Intersection-over-Union), and that LoRA consistently improves in-domain performance. In contrast, training on multiple EM datasets leads to severe performance degradation for all models considered, with only marginal gains from PEFT. Exploration of the latent representation space through various techniques (PCA, Fréchet Dinov2 distance, and linear probes) reveals a pronounced and persistent domain mismatch between the two considered EM datasets in spite of their visual similarity, which is consistent with the observed failure of paired training. These results suggest that, while VFMs can deliver competitive results for EM segmentation within a single domain under lightweight adaptation, current PEFT strategies are insufficient to obtain a single robust model across heterogeneous EM datasets without additional domain-alignment mechanisms.

电子显微镜视觉大模型域迁移分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。