小模型也能逐步推理，揭示思维过程如何提升泛化能力。
Protoreasoning in Tiny Transformers
- 用简化版思维链（原型推理）让百万参数小模型实现分步思考。
- 在嵌套括号任务中，推理路径显著缩小分布外泛化差距。
- 适合研究推理机制本质，尤其关注小模型如何学习通用算法。
我们发现，微型Transformer模型可有效采用一种称为原型推理的简单链式思维形式，使我们在约100万参数规模的模型上研究逐步推理成为可能，为更细致的实验与分析开辟了新路径。当前大语言模型展现出出色的逐步推理能力，但我们仍不清楚其普适性：即模型何时、如何学习真正通用的算法，而非仅依赖启发式经验。这类问题在计算成本高、训练数据不透明的前沿大模型上难以验证。为在远低于自然语言能力阈值的模型规模下开展研究，我们设计了面向推理的任务，基于迪克语言（Dyck语言）——即正确嵌套括号的句子。结果表明，原型推理轨迹显著缩小了分布外泛化差距；消融实验确认，这一提升源于轨迹内容本身，而非仅额外的标记数量。
原文摘要 · Abstract (English)
We show that tiny transformers can profitably employ a simple form of Chain of Thought, which we call protoreasoning, allowing us to study step-by-step reasoning on ~1M-parameter models and opening up opportunities for much more detailed experimentation and analysis than is feasible for larger models. Current Large Language Models exhibit impressive step-by-step reasoning, but we have yet to understand its generality, i.e., when and how LLMs learn genuinely general algorithms rather than "bags of heuristics." Such questions are hard to settle on compute-intensive frontier models trained on opaque data. To work at model scales far below the threshold for natural-language competence, we define reasoning-friendly tasks on Dyck languages (sentences of correctly nested brackets). We find that protoreasoning traces substantially close the out-of-distribution generalization gap, and ablations confirm that the trace's content, not merely its extra tokens, drives the gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。