arXiv:2605.08013cs.AI2026-05被引 1

让命令行智能体在有限观察下高效学习,精准分配行动奖励。

Learning CLI Agents with Structured Action Credit under Selective Observation

论文配图:Learning CLI Agents with Structured Action Credit under Selective Observation
图 1 · 摘自论文原文
  • 引入$σ$-Reveal机制,按预算选择关键上下文信息。
  • 提出$\mathrm{A}^3$方法,基于语法树残差与轨迹边际分配奖励。
  • 构建ShellOps数据集,支持仓库级命令行任务评估。

命令行接口(CLI)智能体正成为与动态文件系统、可执行程序及在线反馈交互的实用范式。现有研究使用强化学习从可验证的任务反馈中学习交互能力,但极少利用CLI动作固有的结构特征作为学习信号。此外,代码智能体面临两大瓶颈:一是在部分观测下从大型代码库中识别相关证据;二是需将稀疏终端奖励分配给影响长多轮轨迹的行动。本文通过壳驱动的信息提取与文件编辑任务研究这些挑战。针对选择性观察,提出$σ$-Reveal,一种推理时的上下文选择机制,仅使用有限令牌预算。针对信用分配,提出行动优势分配($\mathrm{A}^3$),一种保留标准智能体强化学习算法复杂度的原生方法,其利用任务级相对反馈、基于抽象语法树(AST)的动作子链残差和树级轨迹边际构造回合级优势。为进一步评估该问题设置,构建了ShellOps——一个覆盖仓库环境中的命令行任务的可验证数据集套件。

原文摘要 · Abstract (English)

Command line interface (CLI) agents are emerging as a practical paradigm for agent-computer interaction over evolving filesystems, executable command line programs, and online execution feedback. Recent work has used reinforcement learning (RL) to learn these interaction abilities from verifiable task feedback, yet few methods exploit the native structured attributes of CLI actions as learning signals. Beyond this underused action structure, CLI learning also couples two bottlenecks for coding agents. First, the agent must identify task-relevant evidence in a large codebase from partial observations. Second, sparse terminal rewards must be assigned to the actions that shape a long multi-turn trajectory. We study these bottlenecks through shell-driven information extraction and file editing tasks. For selective observation, we introduce $σ$-Reveal, an inference-time mechanism that selects token-budgeted context for the same CLI. For credit assignment, we propose Action Advantage Assignment ($\mathrm{A}^3$), a native agentic RL method that preserves the algorithmic complexity of standard agentic RL. $\mathrm{A}^3$ constructs turn-level advantages from episode-level relative feedback, abstract syntax tree (AST) based action sub-chain residuals, and tree-level trajectory margins. To further evaluate this problem setting, we construct ShellOps, a verifiable dataset suite covering CLI tasks in repository environments.

命令行强化学习智能体结构化动作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。