用隐式动作空间让大模型更可控、更高效地完成任务。
Controlling Large Language Model with Latent Actions
- 引入隐式动作空间，替代传统逐词控制，提升生成多样性。
- 数学推理任务得分42.4，结合搜索算法达68.2，优于基线38.2。
- 保持原始模型能力，计算效率提升50%，适合强化学习应用。
使用强化学习（RL）适配大语言模型（LLMs）到下游任务已被证明是有效方法。然而，LLMs 本身不定义智能体的结构，尤其在动作空间设计上缺乏明确规范。本文研究学习一种紧凑的隐式动作空间，以增强 LLM 的可控制性与探索能力。提出 CoLA 框架，将隐式动作空间集成至预训练的 LLM。在 Llama-3.1-8B 上实验表明，相较于基于词元的动作，CoLA 的隐式动作能产生更高语义多样性。在下游任务中，CoLA 结合 RL 在 math500 基准上取得 42.4 分，超过基线 38.2；结合蒙特卡洛树搜索变体达 68.2。此外，CoLA 在不损害预训练模型能力的前提下，持续提升基于智能体的任务表现。最后，在需要增强思维提示的任务中，计算时间减少 50%。结果表明，CoLA 具有推动基于强化学习的 LLM 适配应用的潜力。
原文摘要 · Abstract (English)
Adapting Large Language Models (LLMs) to downstream tasks using Reinforcement Learning (RL) has proven to be an effective approach. However, LLMs do not inherently define the structure of an agent for RL training, particularly in terms of defining the action space. This paper studies learning a compact latent action space to enhance the controllability and exploration of RL for LLMs. We propose Controlling Large Language Models with Latent Actions (CoLA), a framework that integrates a latent action space into pre-trained LLMs. We apply CoLA to the Llama-3.1-8B model. Our experiments demonstrate that, compared to RL with token-level actions, CoLA's latent action enables greater semantic diversity in text generation. For enhancing downstream tasks, we show that CoLA with RL achieves a score of 42.4 on the math500 benchmark, surpassing the baseline score of 38.2, and reaches 68.2 when augmented with a Monte Carlo Tree Search variant. Furthermore, CoLA with RL consistently improves performance on agent-based tasks without degrading the pre-trained LLM's capabilities, unlike the baseline. Finally, CoLA reduces computation time by half in tasks involving enhanced thinking prompts for LLMs by RL. These results highlight CoLA's potential to advance RL-based adaptation of LLMs for downstream applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。