将大模型压缩成小模型，仍保持通用能力。
Foundry: Distilling 3D Foundation Models for the Edge
- 用超令牌重构教师模型的表征，实现高效压缩。
- 压缩后模型在分类、分割等任务上接近原模型性能。
- 适合机器人、AR/VR等边缘设备部署。
自监督学习预训练的大规模基础模型已成为强大的通用特征提取器，但其庞大体积和高计算成本使其难以部署于机器人、AR/VR头显等边缘设备。现有压缩技术如标准知识蒸馏虽能生成高效‘专用’模型，却牺牲了基础模型关键的下游无关通用性。本文提出基础模型蒸馏（FMD），一种将大规模自监督模型压缩为小型、高效且忠实的代理模型的新范式。我们实现了首个面向3D点云的FMD系统——Foundry。该方法训练学生模型学习一组压缩后的超令牌，以重建教师模型的令牌级表征，捕捉其潜在空间的紧凑基底。单一蒸馏模型在分类、部件分割和少样本场景中均表现出强迁移能力，接近全尺寸基础模型性能，同时显著减少令牌数量与浮点运算量，使模型更适用于资源受限硬件部署。
原文摘要 · Abstract (English)
Foundation models pre-trained with self-supervised learning (SSL) on large-scale datasets have become powerful general-purpose feature extractors. However, their immense size and computational cost make them prohibitive for deployment on edge devices such as robots and AR/VR headsets. Existing compression techniques like standard knowledge distillation create efficient 'specialist' models but sacrifice the crucial, downstream-agnostic generality that makes foundation models so valuable. In this paper, we introduce Foundation Model Distillation (FMD), a new paradigm for compressing large SSL models into compact, efficient, and faithful proxies that retain their general-purpose representational power. We present Foundry, the first implementation of FMD for 3D point clouds. Our approach, Foundry, trains a student to learn a compressed set of SuperTokens that reconstruct the teacher's token-level representations, capturing a compact basis of its latent space. A single distilled model maintains strong transferability across diverse downstream tasks-classification, part segmentation, and few-shot scenarios-approaching full foundation-model performance while using significantly fewer tokens and FLOPs, making such models more practical for deployment on resourceconstrained hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。