让模型自适应决定生成顺序，提升推理与规划能力
Reinforced Context Order Recovery for Adaptive Reasoning and Planning
- 用强化学习从文本中自动学习最优生成顺序
- 在多个推理任务上超越基线，甚至超过已知最优顺序的模型
- 无需人工标注，适合需要灵活逻辑生成的场景
当前因果语言模型和离散扩散模型主要以固定（从左到右）或随机顺序生成文本，这可能偏离原始生成逻辑。本文发现，这类模型在需自适应生成顺序的问题上表现不佳，用$ \mathcal{V}$-信息框架进行了刻画。为此提出基于强化学习的上下文顺序恢复方法ReCOR，无需标注即可从文本数据中提取依赖数据的生成顺序。通过自监督的词预测统计，ReCOR估计每个待填词的预测难度，并在训练和推理时自适应选择下一个词。在多个挑战性推理与规划数据集上的实验表明，ReCOR性能显著优于基线，有时甚至超过使用真实生成顺序监督的“理想”模型。
原文摘要 · Abstract (English)
Modern causal language models, followed by rapid developments in discrete diffusion models, can now produce a wide variety of interesting and useful content. However, these families of models are predominantly trained to output tokens with a fixed (left-to-right) or random order, which may deviate from the logical order in which tokens are generated originally. In this paper, we observe that current causal and diffusion models encounter difficulties in problems that require adaptive token generation orders to solve tractably, which we characterize with the $\mathcal{V}$-information framework. Motivated by this, we propose Reinforced Context Order Recovery (ReCOR), a reinforcement-learning-based framework to extract adaptive, data-dependent token generation orders from text data without annotations. Self-supervised by token prediction statistics, ReCOR estimates the hardness of predicting every unfilled token and adaptively selects the next token during both training and inference. Experiments on challenging reasoning and planning datasets demonstrate the superior performance of ReCOR compared with baselines, sometimes outperforming oracle models supervised with the ground-truth order.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。