用人类示范引导智能体进化,解决长周期任务中反馈稀疏难题
DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations

- 以人类示范替代自生成数据,指导外部架构的精准修改
- 在长周期随机环境中,性能提升37%且错误可追溯
- 适合需要稳定、可解释性优化的复杂任务场景
智能体束带演化通过修改语言模型外部结构来提升其能力,而非更新模型权重。该方法在样本高效快速适应中表现优异:智能体可在不损害基础能力的前提下,通过调整外部结构获得特定任务能力。以往工作表明,自生成轨迹可支持束带搜索,暗示智能体可通过实践习得新技能。然而,在长周期随机环境中,自实践变得脆弱:奖励稀疏、结果方差大、失败难以归因于具体束带机制。本文提出DemoEvolve,一种基于人类示范的束带演化方法。当仅靠奖励搜索过于宽泛且噪声大时,人类示范作为专家经验,为代码生成器提供指导,实现束带层面的可诊断与可编辑。在Liar's Dice上,短周期任务中自演进有效;而在更复杂的长周期随机任务Balatro中,自演进受稀疏反馈和候选选择噪声误导,单纯文本教程无法带来稳定改进。在相同预算下,DemoEvolve生成更有效、可审计的束带修改,性能显著提升。总体而言,示范使稀疏反馈下的束带演化更具可诊断性、可定位性和稳定性。
原文摘要 · Abstract (English)
Agent harness evolution improves frozen language-model agents by modifying the executable structures around them. We study this paradigm as a form of sample-efficient fast adaptation: instead of updating model weights, an agent can acquire task-specific competence by changing its external harness, while leaving the base model's general capabilities intact. Prior work shows that self-generated rollouts can support harness search, suggesting that agents may acquire new task competence through practice. Yet in long-horizon stochastic environments, self-practice becomes fragile: rewards are sparse, outcomes are high-variance, and failures are hard to attribute to concrete harness mechanisms. We introduce DemoEvolve, a demonstration-bootstrapped approach to harness evolution. When reward-only search is too broad and noisy, competent human trajectories serve as expert reference experience for the coding proposer, guiding harness-level diagnosis and editing. Experiments on Liar's Dice show that self-rollout evolution can work when episodes are short and failures are attributable. In contrast, Balatro exposes a harder long-horizon stochastic regime, where self-rollout evolution is misled by sparse feedback and candidate-selection noise, while tutorial-like textual knowledge alone does not yield stable improvement. Under the same limited budget, DemoEvolve produces more effective and auditable harness edits and achieves better performance. Overall, demonstrations make sparse-feedback harness evolution more diagnosable, localizable, and stable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。