小模型生成代码时,训练全程评估比单点评估更真实
Evaluating Tiny Recursive Models Across Training for Code Generation
- 用递归结构构建2800万参数的小模型,重复使用模块增深度
- 训练40轮中模型表现起伏,自由生成时优势在后期显现
- 适合关注小模型泛化与训练动态的研究者和实践者
代码生成日益依赖大规模Transformer模型,但其高昂成本催生了对小型模型的需求,尤其在数据有限场景。递归模型通过重复使用单一模块来增加深度,而非堆叠独立层。现有评估多基于教师强制拟合(使用真实前缀的下一个词损失)或任务准确率,仅在单个检查点进行,而代码实际是模型自主生成的。这种教师强制的优势能否延续至自由生成,以及是否贯穿整个训练过程,尚不明确。为此,我们对比了一个约2800万参数的自回归型微型递归模型(TRM-AR)在自然语言转Python代码任务上的表现,与参数匹配和深度匹配的对照组,追踪40个训练周期、三个随机种子下的拟合与生成能力。参数匹配下,递归模型在拟合、生成和泛化上均优于对照组,恢复了约45%的验证损失差距和57%的生成质量差距,但每步耗时约为对照组的175倍。而在等效深度下,更大规模的Transformer在最优验证点仍表现更优,表明TRM-AR的优势在于抗过拟合,而非更强能力。研究建议应结合拟合与生成,在训练轨迹全程评估递归模型。
原文摘要 · Abstract (English)
Code generation increasingly relies on large transformer models, whose capability advances with scale. Yet such a scale is costly, creating demand for small models, especially where data is limited. Recursive models address this by reusing a single block to add depth rather than stacking independent layers. Such models are typically evaluated by teacher-forced fit (next-token loss on ground-truth prefixes) or task accuracy, at a single checkpoint, whereas code is produced by free-running generation, where the model extends its own output. Whether a teacher-forced advantage survives free-running generation, and whether it holds across training, remains open. To study both, we compare a ~28M-parameter autoregressive Tiny Recursive Model (TRM-AR) on natural-language-to-Python code generation against parameter-matched and depth-matched controls, tracking fit and generation across 40 epochs and three seeds. The fit ranking between the recursive model and the depth-matched control reverses twice. Selecting each checkpoint by validation loss and examining the trajectory yields a consistent comparison. At equal parameters, TRM-AR fits, generates, and generalizes better than the parameter-matched control while recovering approximately 45% of the validation-loss gap and 57% of the generation-quality gap between the two controls, at roughly 175 times the per-step cost of the parameter-matched control. However, at equal effective depth, the larger transformer fits and generates better at its validation optimum, suggesting TRM-AR's advantage lies in resistance to overfitting, not greater capability. These findings suggest that recursive code generation models should be evaluated jointly on fit and generation across the training trajectory rather than at a single checkpoint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。