arXiv:2511.19484cs.SEcs.LG2025-11被引 10

一个简单高效的自监督学习工具库,助力快速探索大模型预训练。

stable-pretraining-v1: Foundation Model Research Made Simple

  • 模块化设计整合了数据增强、评估流程等核心功能
  • 全面日志记录让训练过程可监控可复现
  • 适合希望快速实验的科研人员和工程团队

基础模型与自监督学习(SSL)已成为现代人工智能的核心,但相关研究仍受限于复杂代码库、重复实现以及大规模实验带来的重工程负担。我们提出 stable-pretraining,一个基于 PyTorch、Lightning、Hugging Face 和 TorchMetrics 的模块化、可扩展且性能优化的库。不同于以往仅聚焦于复现顶尖结果的工具包,stable-pretraining 强调灵活性与迭代速度:它统一了探针、崩溃检测指标、数据增强流水线及可扩展评估流程等关键组件,构建出一致可靠的框架。核心设计理念是全程记录,实现对训练动态的细粒度可视性,使调试、监控与复现变得无缝。我们通过实例验证其能力——以极低开销生成新研究洞见,如深度卷积表示探针与合成数据微调下 CLIP 性能退化的分析。该工具库在降低入门门槛的同时支持大规模实验,旨在加速发现,拓展基础模型研究的可能性。

原文摘要 · Abstract (English)

Foundation models and self-supervised learning (SSL) have become central to modern AI, yet research in this area remains hindered by complex codebases, redundant re-implementations, and the heavy engineering burden of scaling experiments. We present stable-pretraining, a modular, extensible, and performance-optimized library built on top of PyTorch, Lightning, Hugging Face, and TorchMetrics. Unlike prior toolkits focused narrowly on reproducing state-of-the-art results, stable-pretraining is designed for flexibility and iteration speed: it unifies essential SSL utilities--including probes, collapse detection metrics, augmentation pipelines, and extensible evaluation routines--within a coherent and reliable framework. A central design principle is logging everything, enabling fine-grained visibility into training dynamics that makes debugging, monitoring, and reproducibility seamless. We validate the library by demonstrating its ability to generate new research insights with minimal overhead, including depthwise representation probing and the analysis of CLIP degradation under synthetic data finetuning. By lowering barriers to entry while remaining scalable to large experiments, stable-pretraining aims to accelerate discovery and expand the possibilities of foundation model research.

自监督学习基础模型工具库可复现

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。