Tiny递归模型在ARC任务中表现优异,实则依赖测试时计算和任务标识符,非深层推理。
Tiny Recursive Models on ARC-AGI-1: Inductive Biases, Identity Conditioning, and Test-Time Compute
- 利用测试时采样与多数投票提升性能,单次推理升11个百分点
- 模型准确率高度依赖任务标识符,替换后直接归零
- 递归更新效果有限,早期步骤已达成主要准确率,适合高效推理场景
Tiny递归模型(TRM)被提出作为解决抽象推理基准(ARC)类任务的参数高效替代方案。原工作报告了出色性能,并认为递归潜在更新支持非平凡推理,但其性能究竟来自架构、测试时计算还是任务先验仍不明确。本文针对ARC-AGI-1上的TRM检查点进行实证分析,发现四类行为特征及效率对比:第一,测试时增强与多数投票占性能提升大头,1000样本投票使Pass@1提升约11个百分点;第二,消融实验显示模型严格依赖任务标识符,替换为空白或随机标记导致准确率为零;第三,递归轨迹分析表明大部分精度在首次递归步骤即达成,性能在少数潜变量更新后即饱和,体现浅层有效递归;第四,早期训练实验显示,强增强策略扩展候选解分布,提升多样本成功率。最后,将TRM与基于Llama 3 8B的朴素QLoRA微调在标准ARC-AGI-1上对比,结果显示TRM的非自回归设计具有更高吞吐量和显著更低内存占用。整体而言,TRM在ARC-AGI-1上的表现源于效率、任务特定条件与激进测试时计算的协同作用,而非深层内部推理。
原文摘要 · Abstract (English)
Tiny Recursive Models (TRM) were proposed as a parameter-efficient alternative to large language models for solving Abstraction and Reasoning Corpus (ARC) style tasks. The original work reports strong performance and suggests that recursive latent updates enable non-trivial reasoning, but it remains unclear how much of this performance stems from architecture, test-time compute, or task-specific priors. In this technical note, we empirically analyze the ARC Prize TRM checkpoint on ARC-AGI-1 and report four behavioral findings and an efficiency comparison. First, we show that test-time augmentation and majority-vote ensembling account for a substantial fraction of reported performance: the 1000-sample voting pipeline improves Pass@1 by about 11 percentage points over single-pass canonical inference. Second, a puzzle-identity ablation reveals strict dependence on task identifiers: replacing the correct puzzle ID with a blank or random token yields zero accuracy. Third, a recursion trajectory analysis shows that most of the final accuracy is achieved at the first recursion step and that performance saturates after few latent updates, indicating shallow effective recursion. Fourth, early-stage training experiments under canonical versus heavy augmentation regimes suggest that heavy augmentation broadens the distribution of candidate solutions and improves multi-sample success. Finally, we compare TRM with a naive QLoRA fine-tune of Llama 3 8B on canonical ARC-AGI-1, finding that TRM's non-autoregressive design achieves much higher throughput and substantially lower memory usage in this setting. Overall, TRM's ARC-AGI-1 performance appears to arise from an interaction between efficiency, task-specific conditioning, and aggressive test-time compute rather than deep internal reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。