发现语言模型内部有分层策略，可自下而上优化提升推理能力
Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies
- 通过残差流分解模型内部策略，揭示分层演进规律
- 早期低层高熵探索，顶层趋于确定性，不同模型结构差异明显
- 提出自下而上的优化方法，显著提升复杂推理任务表现
现有强化学习方法将大语言模型视为单一策略，忽视其内部机制。本文通过Transformer的残差流，将基于LLM的策略分解为内部层策略与模块策略。对内部策略的熵分析揭示两种模式：(1) 通用规律是早期层高熵探索，顶层逐步转为确定性精炼；(2) Qwen表现出明确的渐进式推理结构，而Llama则呈现突变收敛。进一步发现，优化早期层能促进特征精炼，迫使底层提前捕捉高层推理表征。基于此，我们提出自下而上的策略优化（BuPO），在复杂推理基准上通过早期层优化重建模型推理基础，实验验证其有效性。
原文摘要 · Abstract (English)
Existing reinforcement learning (RL) approaches treat large language models (LLMs) as a unified policy, overlooking their internal mechanisms. In this paper, we decompose the LLM-based policy into Internal Layer Policies and Internal Modular Policies via the Transformer's residual stream. Our entropy analysis of internal policy reveals distinct patterns: (1) universally, internal policies evolve from high-entropy exploration in early layers to deterministic refinement in the top layers; and (2) Qwen exhibits an explicit progressive reasoning structure, contrasting with the abrupt convergence in Llama. Furthermore, we discover that optimizing internal layers induces feature refinement, forcing lower layers to capture high-level reasoning representations early. Motivated by these findings, we propose Bottom-up Policy Optimization (BuPO), a novel RL paradigm that reconstructs the LLM's reasoning foundation from the bottom up by optimizing internal layers in early stages. Extensive experiments on complex reasoning benchmarks demonstrate the effectiveness of BuPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。