让AI代理在真实企业场景中按时间回放评估,解决静态测试不真实的问题。
What Could the Agent See at 19:05? Generating Temporal Enterprise Scenarios from Real Research and Replaying Them to Evaluate Agents

- 基于真实研究生成随时间演化的企业场景,支持任意时刻回放。
- 通过差分缓存预计算历史状态,评估速度极快且可复现。
- 适合评估多应用、时序敏感的企业级AI代理,避免未来信息泄露。
企业AI代理在多个应用间协作,其数据持续变化,因此答案的正确性取决于提问时刻可见的数据。当前离线评估仅基于单一静态快照(即任务结束时),只能评价最终状态,而忽略过程中每个时刻都是独立情境,需对应不同正确答案。若为每个时刻重建完整租户环境,成本过高;且单个快照会泄露未来信息,无法体现多应用、时间有序的真实工作流程。本文提出系统:从真实研究中生成具人物角色驱动、随时间演化的企业世界,并可在任意选定时刻回放,以评估任意可插拔的代理。基于模式推断的时间描述,驱动确定性+大模型重建每条记录的历史状态;由于可查询时刻有限,所有重建均预计算为紧凑差分缓存,使评估变为无模型路径的快速查找。我们阐述了系统设计、跨流程架构及早期评估经验。
原文摘要 · Abstract (English)
Enterprise AI agents act across many apps whose data changes continuously, so an answer is correct only relative to what data existed and who could see it at the moment it was asked. Offline evaluation today grades against a single static snapshot, effectively the end of the episode. So, it can only evaluate one situation, the final one, even though every earlier moment of the episode is a different situation that invites its own realistic questions with its own correct answers. Recreating each of those moments as a separate snapshot would mean re-provisioning a whole tenant per instant, which is prohibitively costly; and even a single snapshot leaks future state hidden inside records and cannot represent the multi-app, time-ordered way real work happens. Our system closes two gaps at once: it generates a realistic, persona-driven, temporally-evolving enterprise world from real research, and replays that world at any chosen moment to evaluate any pluggable agent. A schema-inferred temporal description drives a deterministic-plus-LLM rebuild of each record's past state; because the queryable moments are finite, all rebuilds are precomputed into a compact difference cache, making evaluation a fast, reproducible lookup with no model in the path. We describe the design, an architecture spanning both flows, and early experience evaluating enterprise agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。