新混合解码器在代码训练中显著提升记忆保持能力。
SamatNext v0.2-B: An Exploratory Study of RMS-Normalized Hybrid Decoders for Curriculum Retention in Small Code Models
- 交替使用注意力层与简化线性状态混合层，结合RMS归一化增强稳定性。
- 在阶段5测试中达100%通过率，保留98.8%阶段3语义行为。
- 适合关注小模型持续学习与遗忘问题的研究者参考。
标准自回归Transformer解码器在顺序微调时容易出现严重遗忘。本文评估了实验性356M参数混合序列解码器SamatNext v0.2-B，其交替使用类Differential-Attention层与受DeltaNet启发的简化线性状态混合层，并采用RMS归一化与输出缩放校准。在受控的分阶段Python代码课程下，该模型在阶段5留出测试中达到100.0%通过率，保留98.8%的相邻阶段3语义行为，阶段2E早期语法测试得分为12.0%。最强的参数匹配Transformer基线在阶段5达97.6%，但仅保留6.0%的阶段3行为。两者在长时早阶段记忆上仍表现薄弱，结果表明该架构在可控环境下改变了记忆/可塑性权衡，而非普遍解决灾难性遗忘。代码、模型规格、评估脚本及结果表格均已提供以供独立验证。
原文摘要 · Abstract (English)
Standard autoregressive Transformer decoders can often exhibit substantial forgetting under sequential fine-tuning on shifting curriculum distributions. This technical report evaluates SamatNext v0.2-B, an experimental 356M-parameter hybrid sequence decoder that alternates Differential-Attention-style layers with DeltaNet-inspired simplified linear-state mixer layers using RMS normalization and output scale calibration. We study the model under a controlled staged Python code curriculum and compare it with a parameter-matched Transformer baseline. In this setting, SamatNext v0.2-B achieves a 100.0% pass rate on the controlled Stage 5 holdout while retaining 98.8% of adjacent Stage 3 semantic behavior and reaching 12.0% on the Stage 2E early syntax holdout. The strongest Transformer baseline reaches 97.6% on Stage 5 but retains only 6.0% of Stage 3 behavior. Both architectures remain weak on long-horizon early-stage retention, so the result should be interpreted as evidence of an altered retention/plasticity tradeoff in this controlled setting, not as a general solution to catastrophic forgetting. Code, model specifications, evaluation scripts, and result tables are provided for independent verification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。