评估智能体时,何时才算真正结束?关键看结果是否确定且不被干扰。
When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation
- 通过延迟完成机制判断结果是否最终确定
- 实验发现延迟写入会改变后续运行分数,除非状态隔离
- 建议记录未决操作影响,确保评估公平可复现
智能体评估通常以运行终止时的状态打分,并计为一次独立试验。将该得分视为独立试验的最终结果,需满足结果终局性与跨单元分离性。结果终局性要求后续事件无法改变已宣称的结果,跨单元分离性要求早期运行不会影响后续运行的相关条件。终点本身并不保证这两项条件,二者可独立成立。等待延迟结果可能使标签最终确定,即使状态仍对其他运行可用;隔离可能防止影响传递,即使结果尚未明确。我们提出一种完成性论证,识别每项决策所需证据:只有当所有相关影响均被解决或紧密约束时,最终成功或失败标签才成立,任何剩余不确定性必须报告。首先,在固定动作的可控重放中,我们发现非零延迟操作下终点与终态标签总不一致;延迟写入在共享状态下会改变下一运行得分,但在命名空间或验证重置后则无此影响。其次,在对十项公开协议的审查中,显式记录重置或有意保留的情况多于未完成操作或独立评分的证据。最后,我们提出开放效应记录机制,追踪可能影响终点后评分或其它运行的操作与资源状态及其潜在影响。
原文摘要 · Abstract (English)
Agent evaluations commonly score the state observed when a run stops and count the run as one trial. Interpreting that score as a final result from a separate trial requires outcome finality and cross-unit separation. Outcome finality requires that later events cannot change the claimed result, while cross-unit separation requires that earlier runs cannot change the relevant conditions of later ones. The endpoint establishes neither condition by itself, and the two can hold independently. Waiting for a delayed outcome may settle the label even though its state remains available to another run. Isolation may prevent carryover even though the scored outcome remains unresolved. We develop a completion argument that identifies the evidence needed for each decision. A final success or failure label is justified only when every relevant effect is resolved or bounded tightly enough to fix the outcome. Any remaining uncertainty must be reported. First, in a controlled replay with fixed agent actions, we find that endpoint and terminal labels differ for every nonzero-delay operation and that a delayed write changes the next run's score under shared state but has no such effect after namespacing or verified reset. Second, in a review of ten public protocols, we find that reset or deliberate retention is documented explicitly more often than unfinished operations or evidence for separate scoring. Finally, we propose an open-effects record for operations and resources that may remain relevant after the endpoint, their status, and their possible effects on the scored outcome or another run.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。