无需额外参数即可高效微调3D模型,避免过拟合且保持推理速度。
An Adapter-free Fine-tuning Approach for Tuning 3D Foundation Models
- 只微调部分编码器,通过动量一致性约束保护通用特征。
- 5次采样下比之前方法提升3.30%,半监督时最高增益达6.13%。
- 无额外参数,适合低资源设备部署,适合小样本3D任务研究者。
点云基础模型展现出强大泛化能力,但在数据稀缺场景下适配下游任务仍具挑战。全量微调易导致过拟合并引发预训练表征显著偏移,而现有参数高效微调(PEFT)方法虽缓解此问题,却需引入额外可训练组件,增加推理延迟。本文提出动量一致性微调(MCFT),一种无需适配器的微调方法,平衡全量与参数高效微调之间的差距。MCFT仅选择性微调预训练编码器的一部分,并施加基于动量的一致性约束,以保留任务无关表征。与PEFT不同,MCFT不引入额外表示学习参数,维持原始模型参数量与推理效率。进一步提出两种变体:利用大量无标签数据提升少样本性能的半监督框架,以及通过结构化层剪枝提升计算效率的剪枝版本。在物体识别与部件分割基准上的大量实验表明,MCFT持续优于先前方法,在5次采样设置下实现3.30%的提升,半监督学习下最高达6.13%的改进,同时仍适用于资源受限部署。
原文摘要 · Abstract (English)
Point cloud foundation models demonstrate strong generalization, yet adapting them to downstream tasks remains challenging in low-data regimes. Full fine-tuning often leads to overfitting and significant drift from pre-trained representations, while existing parameter-efficient fine-tuning (PEFT) methods mitigate this issue by introducing additional trainable components at the cost of increased inference-time latency. We propose Momentum-Consistency Fine-Tuning (MCFT), an adapter-free approach that bridges the gap between full and parameter-efficient fine-tuning. MCFT selectively fine-tunes a portion of the pre-trained encoder while enforcing a momentum-based consistency constraint to preserve task-agnostic representations. Unlike PEFT methods, MCFT introduces no additional representation learning parameters beyond a standard task head, maintaining the original model's parameter count and inference efficiency. We further extend MCFT with two variants: a semi-supervised framework that leverages abundant unlabeled data to enhance few-shot performance, and a pruning-based variant that improves computational efficiency through structured layer removal. Extensive experiments on object recognition and part segmentation benchmarks demonstrate that MCFT consistently outperforms prior methods, achieving a 3.30% gain in 5-shot settings and up to a 6.13% improvement with semi-supervised learning, while remaining well-suited for resource-constrained deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。