小模型赛时微调，用极低算力提升逻辑推理能力
Test-time Adaptation of Tiny Recursive Models
- 用预训练小模型在竞赛任务上直接全量微调
- 赛时仅12.5k步梯度更新，得分从10%提升至6.67%
- 适合资源受限但需快速适应新任务的场景
在2025年ARC Prize竞赛临近结束时，主流开源方法——称为TRM（Tiny Recursive Models）——采用一个700万参数的递归神经网络，在增强版ARC任务上进行训练，于公开的ARC AGI II评估集上获得约7.8%的分数，但所需算力远超竞赛允许范围。本文表明，若从已在公开ARC任务上预训练的小型递归模型出发，可在竞赛允许的算力范围内高效微调。具体而言，该模型在48小时内使用4块H100 SXM GPU对1,280个公开任务进行了超过70万次优化器步数的预训练，于公开评估集上取得约10%的分数。随后，在竞赛期间仅用12,500次梯度更新进行后训练，便在半私有评估任务上达到6.67%的得分。值得注意的是，该性能通过完全微调小型模型实现，而非仅微调LoRA或任务嵌入。
原文摘要 · Abstract (English)
Prior to the close of the 2025 ARC Prize competition, the leading open source approach - known as TRM, or Tiny Recursive Models - involved training a 7M parameter recursive neural network on augmented variants of ARC tasks. That approach scored approximately 7.8% on the public ARC AGI II evaluation set, but required a level of compute far in excess of what is allowed during the competition. This paper shows that, by starting from a tiny recursive model that has been pre-trained on public ARC tasks, one can efficiently fine-tune on competition tasks within the allowed compute limits. Specifically, a model was pre-trained on 1,280 public tasks for 700k+ optimizer steps over 48 hours on 4xH100 SXM GPUs to obtain a ~10% score on the public evaluation set. That model was then post-trained in just 12,500 gradient steps during the competition to reach a score of 6.67% on semi-private evaluation tasks. Notably, such post-training performance is achieved by full-fine tuning of the tiny model, not LoRA fine-tuning or fine-tuning of task embeddings alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。