首次将存储占用纳入大模型智能体评估，揭示其隐性数据足迹。
The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation
- 设计AgentFootprint基准，量化智能体运行后的持久化存储开销。
- 相同任务下存储量差15.7倍，且与准确率无相关性。
- 采用内容寻址存储可减少4.8至32.7倍存储，同时保证可恢复性。
当前大模型智能体评估关注任务完成度、可靠性与推理成本，却忽略运行后留在磁盘的持久数据，如日志、上下文快照、检查点和调试痕迹。本文提出AgentFootprint，一个跨框架的智能体存储足迹基准。其序列化感知的度量体系涵盖总保留量、通道构成、重复度、增长趋势、可压缩性及对话历史重建能力。该研究揭示测量陷阱：仅用字节级统计会低估重复数据达一个数量级，因数据库分页和JSON转义掩盖了内容重复。通过固定轨迹控制，对比七种持久化框架发现存储量相差6.7倍。在相同模型、工具与任务下，100%准确率配置间存储量差异达15.7倍，且默认配置支持不同的恢复与审计能力。三种完整历史配置在重复观测压力任务中呈现超线性增长。108个经实例归一化的SWE-bench Verified提交轨迹，每实例存储量跨度达三个数量级，且与解决率无显著相关性。内容寻址存储可减少4.8至32.7倍存储，同时保持所有重建得分。结果表明，持久存储应作为与准确率和重建能力并列的基础资源指标报告。
原文摘要 · Abstract (English)
LLM agent benchmarks measure task completion, reliability, and inference cost, but not the persistent data an agent run leaves on disk, including logs, context snapshots, checkpoints, and debug traces. We introduce AgentFootprint, a cross-framework benchmark of post-run agent storage footprint. Its serialization-aware metric suite measures total retention, channel composition, duplication, growth, compressibility, and conversation-history reconstructability. It addresses a measurement trap: naive byte-level measurement understates duplication by an order of magnitude because database paging and JSON escaping obscure repeated content. A fixed-trace control separates agent-generated logical volume from persistence-layer amplification: replaying the same trajectory through seven persisting frameworks yields a 6.7x spread. Under identical models, tools, and tasks, configurations with 100% accuracy differ by 15.7x in retained bytes, although their defaults support different recovery and audit capabilities. Three full-history configurations grow superlinearly on a repeated-observation stress task. Exported trajectories from 108 instance-normalized SWE-bench Verified submissions span three orders of magnitude per instance, with no detectable correlation with resolve rate. A content-addressed store reduces retention by 4.8x-32.7x while preserving every reconstructability score. These results establish persistent storage as a resource metric to report jointly with accuracy and reconstructability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。