让概率化Transformer更易扩展，小模型调好的参数能直接用在大模型上
Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer
- 基于muP思想重标定参数，实现跨规模超参迁移
- 成功将模型扩展至0.4B参数，相同参数量下性能优于标准Transformer
- 适合关注可扩展概率模型的科研与工程人员
概率化Transformer（PT）作为一种白盒概率模型，在小模型和中小数据集上表现出与标准Transformer相似的计算结构和下游任务性能。然而，其对超参数选择更敏感，难以高效扩展。本文借鉴最大更新参数化（muP）思想，对PT进行参数重标定，使在小模型上优化的超参数可直接迁移到大模型，无需额外调参。通过该方法，我们成功将PT扩展至0.4B参数规模。实验表明，在相同参数预算下，PT在掩码语言建模（MLM）任务中持续优于标准Transformer。本工作有望推动概率模型在未来更大规模下的实际应用。
原文摘要 · Abstract (English)
Probabilistic Transformer (PT), a white-box probabilistic model for contextual word representation, has demonstrated substantial similarity to standard Transformers in both computational structure and downstream task performance on small models and small to medium sized datasets. However, PT is less robust to hyperparameter choices than standard Transformers, making it harder to scale efficiently. In this work, we follow Maximal Update Parametrization (muP) to rescale PT's parameters, so that hyperparameters optimized on small models can be transferred to larger models without additional tuning. With this approach, we successfully scale PT to models with up to 0.4B parameters. Experiments show that PT consistently outperforms standard transformer under the same parameter budget on Masked Language Modeling (MLM) tasks. We hope this work will contribute to the practical deployment of probabilistic models at substantially larger scales in the future.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。