构建动态评估环境，精准衡量大模型智能体的自我进化能力
SEAGym: An Evaluation Environment for Self-Evolving LLM Agents
- 设计SEAGym环境，支持多阶段任务追踪与更新验证
- 发现频繁更新未必提升泛化性能，中间快照可能后期失效
- 适合研究智能体自进化机制或评估工具链的开发者
基于大模型的自进化智能体主要通过调整其代理框架（agent harness）实现改进，包括提示、记忆、工具、中间件、运行时状态及模型-工具交互循环。现有评估常简化为孤立任务得分或单一序列曲线，难以判断更新是否带来可复用提升、是否过拟合近期任务、是否增加开销或损害旧有行为。本文提出SEAGym，一个用于评估代理框架更新的动态环境，涵盖训练、验证、测试、回放和成本记录。该环境将兼容Harbor的基准转化为动态自进化任务源，支持训练批次、冻结更新验证、保留的ID与OOD迁移视角、回放诊断及快照与指标记录。在Terminal-Bench 2.0和HLE上实例化SEAGym，对比ACE、TF-GRPO与AHE在统一周期/批次协议下的表现。结果表明，这些评估视图提供了互补信号：频繁更新可能无法提升保留任务性能，有用的中间快照可能后续崩溃，源数据多样性与模型后端会影响框架可靠性。
原文摘要 · Abstract (English)
Self-evolving LLM-based agents improve mainly by changing their agent harness: the structured execution layer around a base model, including prompts, memory, tools, middleware, runtime state, and the model-tool interaction loop. Existing evaluations often reduce this process to isolated task scores or a single sequential curve, obscuring whether an update produces reusable improvement, overfits recent tasks, increases cost, or harms older behavior. We introduce SEAGym, an evaluation environment for measuring agent harness updates across training, validation, test, replay, and cost records. SEAGym turns Harbor-compatible benchmarks into dynamic self-evolution task sources with train batches, frozen update-validation, held-out ID and OOD transfer views, replay diagnostics, and saved snapshot and metric records. Instantiating SEAGym on Terminal-Bench 2.0 and HLE, we compare ACE, TF-GRPO, and AHE under a shared epoch/batch protocol. The results show that these evaluation views provide complementary signals about the evolution process: frequent updates may fail to improve held-out performance, useful intermediate snapshots may collapse later, and source diversity and model backend can affect harness reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。