通过用户迭代优化模型输出数据，让大模型自动提升规划能力。
Iterative Deployment Improves Planning Skills in LLMs
- 用前序模型的输出数据由用户精心筛选后微调新模型，形成闭环迭代。
- 后期模型能生成比初期长得多的计划，展现出涌现的泛化能力。
- 无需显式奖励信号，即可实现类似强化学习的效果，适合安全敏感场景。
我们证明，通过将大型语言模型（LLMs）进行迭代部署，每次使用用户从先前模型输出中精心筛选的数据进行微调，可以显著改变最终模型的特性。在多个规划任务领域测试该机制后，观察到规划能力大幅提升，后期模型展现出涌现的泛化能力，能生成远超初始模型长度的计划。随后的理论分析表明，这种迭代部署实际上在外部循环中实现了强化学习（RL）训练，具有隐式奖励函数。该联系带来两个重要启示：首先，对人工智能安全而言，由于重复部署所隐含的奖励函数未被明确定义，可能对未来模型部署产生意外影响；其次，该机制可被视为一种替代显式强化学习的训练范式，依赖数据筛选而非显式奖励信号。
原文摘要 · Abstract (English)
We show that iterative deployment of large language models (LLMs), each fine-tuned on data carefully curated by users from the previous models' deployment, can significantly change the properties of the resultant models. By testing this mechanism on various planning domains, we observe substantial improvements in planning skills, with later models displaying emergent generalization by discovering much longer plans than the initial models. We then provide theoretical analysis showing that iterative deployment effectively implements reinforcement learning (RL) training in the outer-loop (i.e. not as part of intentional model training), with an implicit reward function. The connection to RL has two important implications: first, for the field of AI safety, as the reward function entailed by repeated deployment is not defined explicitly, and could have unexpected implications to the properties of future model deployments. Second, the mechanism highlighted here can be viewed as an alternative training regime to explicit RL, relying on data curation rather than explicit rewards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。