小模型通过自我验证反思，也能实现大模型级推理效果。
Self-Verifying Reflection Helps Transformers with CoT Reasoning
- 用无自然语言的小模型设计自验证反思框架，提升推理可靠性。
- 仅数百万参数的模型在乘法和数独任务中达到大模型水平性能。
- 适合研究推理机制、低资源场景下的模型优化与可解释性分析。
大型语言模型常在思维链（CoT）推理中进行自我反思，验证当前解的正确性并探索替代方案。然而，近期研究发现模型在CoT中检测错误能力有限，反思如何带来实际性能提升仍不明确。为此，本文提出一种极简推理框架，为小型Transformer模型提供基础的自验证反思机制，确保分析清晰且降低实验成本。理论上，我们证明当验证错误被合理控制时，自验证反思能保证性能提升。实验表明，仅含数百万参数的微型模型在训练和反思执行中均受益，其在整数乘法和数独任务中的表现达到大模型水平。类似大模型结果，强化学习（RL）提升了分布内性能，并激励频繁反思，但主要优化浅层统计模式，未真正减少验证错误。结论是，生成式变压器结合判别式验证，天然促进CoT推理，无论模型规模或是否使用自然语言。
原文摘要 · Abstract (English)
Advanced large language models (LLMs) frequently reflect in reasoning chain-of-thoughts (CoTs), where they self-verify the correctness of current solutions and explore alternatives. However, given recent findings that LLMs detect limited errors in CoTs, how reflection contributes to empirical improvements remains unclear. To analyze this issue, in this paper, we present a minimalistic reasoning framework to support basic self-verifying reflection for small transformers without natural language, which ensures analytic clarity and reduces the cost of comprehensive experiments. Theoretically, we prove that self-verifying reflection guarantees improvements if verification errors are properly bounded. Experimentally, we show that tiny transformers, with only a few million parameters, benefit from self-verification in both training and reflective execution, reaching remarkable LLM-level performance in integer multiplication and Sudoku. Similar to LLM results, we find that reinforcement learning (RL) improves in-distribution performance and incentivizes frequent reflection for tiny transformers, yet RL mainly optimizes shallow statistical patterns without faithfully reducing verification errors. In conclusion, integrating generative transformers with discriminative verification inherently facilitates CoT reasoning, regardless of scaling and natural language.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。