将大模型知识蒸馏到小模型,实现高效农田分割。
JEDI: JEPA-to-Edge Distillation for Efficient Cropland Segmentation from Satellite Imagery

- 用跨架构投影与空间对齐,让小模型学大模型的特征表示。
- 小模型仅404万参数,达到68.0 mIoU,接近大模型70.0的性能。
- 适合资源受限的卫星或田间边缘设备部署。
大型视觉模型为遥感分割提供有用表征,但部署成本过高。现有特征级蒸馏方法通常假设教师与学生架构相似,且在任务训练开始后停止特征对齐。我们提出JEDI(JEPA-to-Edge Distillation),一种两阶段框架,将大型I-JEPA Vision Transformer教师模型的表示迁移到紧凑的SegFormer学生模型。首先,通过跨架构投影和空间对齐,使学生终端表示与教师的令牌空间对齐;随后,在任务适应过程中联合优化监督分割、温度缩放响应蒸馏和持续特征对齐。在CalCROP21数据集上,JEDI-B0仅用404万参数即达68.0 mIoU,较独立学生提升16.0点,仅比6.39亿参数教师模型的70.0 mIoU低2.0点。我们评估了参数量分别为404万、1433万和2800万的SegFormer B0、B1、B2学生模型。在所有变体中,JEDI均显著优于响应、结构、通道及关系蒸馏基线。结果表明,在极端压缩下持续表征对齐极为重要,可大幅降低模型规模与计算量,同时保持分割性能。
原文摘要 · Abstract (English)
Large vision models provide useful representations for remote-sensing segmentation but are often too expensive for deployment at the satellite or field edge. Existing feature-level distillation methods also tend to assume similar teacher and student architectures and often stop feature alignment when task training begins. We introduce JEDI (JEPA-to-Edge Distillation), a two-stage framework that transfers representations from a large I-JEPA Vision Transformer teacher to a compact SegFormer student. First, JEDI aligns the student's terminal representation with the teacher's token space using cross-architecture projection and spatial alignment. It then jointly optimizes supervised segmentation, temperature-scaled response distillation, and persistent feature alignment throughout task adaptation. On CalCROP21, JEDI-B0 achieves 68.0 mean Intersection-over-Union (mIoU) with 4.04M parameters, improving over the standalone student by 16.0 points and coming within 2.0 points of the 70.0 mIoU achieved by the 639M-parameter teacher. We evaluate SegFormer B0, B1, and B2 students with 4.04M, 14.33M, and 28M parameters, respectively. Across all three variants, JEDI consistently outperforms response-, structure-, channel-, and relational-distillation baselines under the same teacher-student setting. These results show that persistent representation alignment is especially valuable under aggressive compression, substantially reducing model size and computation while preserving segmentation performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。