20个新架构修改中仅2个有效，验证了大模型时代仍难迁移。
Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor
- 在1.2B和3B规模下严格控制条件测试20个新修改
- 多数修改无法提升下游任务表现，仅2个在1.2B通过显著性检验
- 强调需噪声基线与跨规模稳定性测试，避免误判
Narang等（2021）在T5-base规模评估40多个Transformer修改后发现多数不具迁移性。五年后，主流模型规模已进入1-3B，下游评估取代预训练困惑度，且新修改方法大量涌现。本文在1.2B和3B规模下，对20个2021年后提出的修改进行严格同数据、同计算、同训练配方控制，并引入多种子基准噪声基线与CLIMB-12作为主评价指标。结果复现原结论：多数修改仍无法迁移。20个修改中仅2个在1.2B通过贝叶斯校正；其中1个在3B下训练不稳定。此外，注意力输出类修改的损失-下游差距扩大数倍：两个显著失败案例验证损失仅比基线低2-3%，但下游得分下降6-16 CLIMB点。结论指出，噪声基线报告、下游评估与跨尺度稳定性测试已成为1-3B模型架构比较的必备条件。
原文摘要 · Abstract (English)
Narang et al. (2021) evaluated 40+ Transformer modifications at T5-base scale and concluded that most did not transfer. Five years later, the typical working regime has moved to 1-3B parameters, downstream evaluation has replaced pretraining perplexity, and a substantially different catalogue of modifications has emerged. We revisit their question by testing 20 post-2021 Transformer modifications at 1.2B and 3B under strict iso-data, iso-compute, iso-recipe control, with a multi-seed baseline noise floor and CLIMB-12 downstream evaluation as the primary metric. The central finding reproduces theirs at this curated set: most modifications do not transfer. Of the 20 modifications, only two clear Bonferroni correction at 1.2B; one of those two further fails to train stably at 3B under the shared recipe. We also find that the loss-downstream gap reported by Tay et al. (2023) enlarges several-fold for attention-output modifications: two significant failures converge to within 2-3% of baseline validation loss yet drop 6-16 CLIMB-points. We conclude that noise-floor reporting, downstream evaluation, and cross-scale stability testing are now prerequisites for architecture comparisons at 1-3B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。