让智能体自我改进过程可审计,通过事件日志实现失败追踪与修复验证。
Regimes: An Auditable, Held-Out-Gated Improvement Loop Demonstrated on LongMemEval with ActiveGraph

- 用事件溯源构建可重放的智能体运行时,修复操作仅作用于指定流水线节点。
- 在LongMemEval上四次提升准确率0.05~0.10,一次提升0.01,部分结果显著。
- 适合关注可解释性、可信自动优化与系统可审计性的研究者与工程团队。
自主改进循环难以信赖,因改进过程常为外部附加:故障未记录,诊断无法复现,升迁或丢弃决策存于独立数据库。我们展示事件溯源的智能体运行时可消除此摩擦,使受控改进成为核心工作流。当智能体状态是只追加事件日志的确定性投影时,故障被记录,运行可精确重放,候选修复局限于类型化的流水线节点,门控可审计,每次升迁或丢弃自身即为事件。我们以Regimes为例,在ActiveGraph运行时上实现故障诊断、修复建议提出,并经静态检查、沙箱执行、样本内评估及保留集验证后才批准。该循环目标无关:同一控制流通过通用接口适配不同任务。在LongMemEval-S上,主要失败并非检索,而是整合后的证据未能正确理解。在五个种子保留拆分中,Regimes发现读者提示修复,使保留集准确率提升0.05至0.10(四次),一次提升0.01(过度推广拆分);两个拆分单独显著(种子5未校正其序列推广结构),总体统计描述性,因各拆分共享一个500题池。持久贡献包括ActiveGraph作为可审计基础,支持保留集门控的改进环,失败模式分类法引导问题定位(其边际价值是主要开放问题),以及提示作为发现探针的假设。
原文摘要 · Abstract (English)
Autonomous improvement loops are hard to trust because the improvement process is usually external scaffolding bolted onto the agent: failures go unlogged, diagnoses cannot be replayed, and promote-or-discard decisions land in a side database rather than the agent's own history. We show that an event-sourced agent runtime removes that friction and turns controlled improvement into a first-class workflow. When the agent's state is a deterministic projection of an append-only event log, failures are recorded, a run replays exactly from its log, candidate patches scope to typed pipeline seams, gates are auditable, and every promotion or discard is itself an event. We demonstrate this with Regimes, a loop on the ActiveGraph runtime that diagnoses failed evaluations, proposes a repair at a pipeline point, and promotes it only after static checks, sandbox execution, in-sample evaluation, and held-out validation. The loop is target-agnostic: the same control flow runs against different tasks through a common interface. On LongMemEval-S the dominant failure is not retrieval but reconciliation: the evidence is already in the assembled context, yet the reader answers incorrectly. Across five seeded held-out splits, Regimes discovers reader-prompt repairs that improve final held-out accuracy by +0.05 to +0.10 in four splits and +0.01 in one over-promotion split; two splits are individually significant (seed 5 unadjusted for its sequential promotion structure), and the pooled count is descriptive only, since the splits share one 500-question pool. The durable contributions are ActiveGraph as an auditable substrate that makes controlled improvement loops tractable, the held-out-gated loop it supports, the failure-regime taxonomy routing each failure to a pipeline location (whose marginal value over an unrouted baseline is the primary open question), and the prompt-as-discovery-probe hypothesis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。