让大模型动态深化关键信息思考，用更少参数实现更强推理。
Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking
- 通过动态路由和残差思维连接，让关键词逐步深化处理。
- 162M模型达到466M模型96.5%性能，训练数据减少43.2%。
- 适合资源受限但需强推理的场景，如移动端部署。
大语言模型在参数受限下处理复杂推理任务时存在性能瓶颈。实证分析发现，难处理的关键词会引发各层梯度突变，暴露标准Transformer的结构弱点。为此，我们提出内省变换器（ITT），将层计算重构成隐式思考步骤。ITT通过自适应令牌路由动态分配计算，利用残差思维连接迭代优化表征，并以思考步编码区分推理阶段。该方法在不增加参数的情况下，深化关键令牌的处理。在162M至466M参数模型上的评估显示，ITT仅用162M参数即达到466M Transformer 96.5%的性能，训练数据需求降低43.2%，并在11项基准测试中优于Transformer与环路变体。通过推理时弹性分配计算，ITT实现了性能与效率的架构感知优化。
原文摘要 · Abstract (English)
Large language models (LLMs) face inherent performance bottlenecks under parameter constraints, particularly in processing critical tokens that demand complex reasoning. Empirical analysis reveals challenging tokens induce abrupt gradient spikes across layers, exposing architectural stress points in standard Transformers. Building on this insight, we propose Inner Thinking Transformer (ITT), which reimagines layer computations as implicit thinking steps. ITT dynamically allocates computation through Adaptive Token Routing, iteratively refines representations via Residual Thinking Connections, and distinguishes reasoning phases using Thinking Step Encoding. ITT enables deeper processing of critical tokens without parameter expansion. Evaluations across 162M-466M parameter models show ITT achieves 96.5\% performance of a 466M Transformer using only 162M parameters, reduces training data by 43.2\%, and outperforms Transformer/Loop variants in 11 benchmarks. By enabling elastic computation allocation during inference, ITT balances performance and efficiency through architecture-aware optimization of implicit thinking pathways.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。