揭示长序列任务中信用分配的方差根源，提出关键转折点与局部信用机制。
The Variance of Thought: Policy Variance, Critical Forks, and Local Credit Assignment
- 用策略方差衡量决策不确定性，识别出影响回报的关键转折点。
- 在双分支场景下，获取优势动作需至少 $Ω(c^2/σ_π^2)$ 次采样，方差越小成本越高。
- 建议对数价值参数化以提升估计效率，适合研究强化学习与推理模型的读者。
长时序语言模型任务——包括多步推理和工具使用智能体——受限于信用分配问题。本文通过策略方差 $σ_π^2(s)=\operatorname{Var}_{a\simπ}[Q_π(s,a)]$ 分析该问题，发现在确定性马尔可夫决策过程（MDP）中，此方差是回报方差的唯一来源，并在特定状态（称为关键转折点）以离散脉冲形式注入。三个结论如下：(i) 策略方差即为发现预算：观察到优势值为 $c$ 的动作需 $Ω(c^2/σ_π^2(s))$ 次采样，该界限在经典双点分支中精确成立；(ii) 策略方差受策略吉尼离散度限制，满足 $σ_π^2(s)\le 1-\|π(\cdot|s)\|_2^2$，该条件无需回放即可从逻辑值计算，是临界性的必要条件；(iii) 剩余时序决定估计成本：在下游成功概率为 $P$ 的分支点，蒙特卡洛优势估计的信噪比约为 $\sqrt{P}$，样本成本随 $1/P$ 增长，分支采样亦共享此代价；而自举法可通过将生存概率乘积转为求和，消除该成本，前提是价值表示具有乘法准确性，从而支持对数价值参数化。
原文摘要 · Abstract (English)
Long-horizon language-model tasks --- multi-step reasoning and tool-using agents alike --- are limited by credit assignment. We analyze it through the policy variance $σ_π^2(s)=\operatorname{Var}_{a\simπ}[Q_π(s,a)]$, which in a deterministic MDP is the sole source of return variance and is injected in discrete pulses at states we call critical forks. Three results follow. (i) Policy variance is a discovery budget: observing an action of advantage $c$ requires $Ω(c^2/σ_π^2(s))$ draws, a bound that is exact on the canonical two-point fork. (ii) Policy variance is bounded by the policy's Gini dispersion, $σ_π^2(s)\le 1-\|π(\cdot|s)\|_2^2$, a rollout-free necessary condition for criticality computable from logits alone. (iii) The remaining horizon sets the estimation cost: at a fork whose downstream success probability is $P$, the Monte Carlo advantage estimate has signal-to-noise ratio of order $\sqrt{P}$, so its sample cost scales as $1/P$ --- a cost that branched sampling shares. Bootstrapping removes it by converting a product of survival probabilities into a sum, provided the value representation is multiplicatively accurate, which argues for log-value parameterization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。