探究生成器访问方式对自回归后训练的影响，发现控制前缀能显著提升性能。
The Role of Generator Access in Autoregressive Post-Training
- 区分从根开始生成和重用已有前缀两种访问方式
- 重用前缀可让条件采样等方法超越单一最优选择
- 接口改变导致KL正则化训练结果出现指数级差距
我们研究了生成器访问方式如何限制自回归后训练。核心问题是学习者是否仅限于从根开始的新序列生成，还是可以返回已构建的前缀并查询其下一个词规则。在根开始模式下，输出采样、生成词的对数概率、top-k报告以及采样路径上的完整下一个词分布均归约为单一实验，受限于到达信息丰富前缀的在线策略概率。弱前缀控制打破了这一限制；一旦获得控制权，更丰富的观测如条件采样或对数几率即可优于top-1访问。仅改变生成器接口，便在KL正则化的结果奖励后训练中造成指数级差距。
原文摘要 · Abstract (English)
We study how generator access constrains autoregressive post-training. The central question is whether the learner is confined to fresh root-start rollouts or can return to previously built prefixes and query the next-token rule there. In the root-start regime, output sampling, generated-token log probabilities, top-$k$ reports, and full next-token distributions along sampled trajectories all reduce to one canonical experiment, limited by the on-policy probability of reaching informative prefixes. Weak prefix control breaks this barrier, and once control is available, richer observations such as conditional sampling or logits can outperform top-$1$ access. Changing only the generator interface creates an exponential gap for KL-regularized outcome-reward post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。