用数据库写时复制技术，精准评估大模型在真实应用中的表现
Copy-on-Write Scoring: Application-Specific Agent Evaluations
- 通过PostgreSQL写时复制机制，在真实应用环境里隔离评估模型操作
- 可定位到具体会话和操作级别的成功与失败，提升调试精度
- 适合开发者快速迭代智能工具界面，尤其对项目管理类应用有效
在软件系统中可信部署基于大语言模型的智能体，需要对特定应用场景的工作流进行细粒度评估，以准确定位其成功与失败环节。然而现有评估方法存在局限：基准测试对特定应用工作流的构念效度低，而复现环境成本高且易发生漂移。本文提出写时复制（Copy-on-Write, CoW）评分框架，利用PostgreSQL层级的写时复制机制，在真实应用环境中直接评估智能体操作，隔离其写入行为。CoW评分生成会话级与操作级评分，清晰揭示智能体在特定应用环境下数据库写入操作的成功与失败位置，支持低成本、高精度的智能体套件与工具界面迭代。我们在开源项目管理平台Plane上验证该框架，分析发现了工具界面的具体问题，并通过对应修复使相关模型性能实现可测量提升。Python库：https://github.com/trail-ml/agent-cow-python
原文摘要 · Abstract (English)
Trustworthy deployment of LLM-based agents in software systems requires evaluating how they perform on application-specific workflows, with enough granularity to localize where they succeed and fail. Yet existing agent evaluation mechanisms are limited: benchmarks have low construct validity for application-specific workflows and environments, and replica evaluation environments are expensive and prone to drift. We propose Copy-on-Write (CoW) Scoring, a framework that evaluates agent operations directly within application environments using a PostgreSQL-level Copy-on-Write mechanism to isolate agent writes. CoW Scoring produces session- and operation-level scores that highlight where agents' database write operations succeed and fail in a given application environment, enabling inexpensive evaluation and iteration on agent harnesses and tool surfaces. We demonstrate the framework on Plane, an open-source project-management platform, where analysis surfaced specific issues in the tool surface, and corresponding fixes produced measurable improvements on affected models. Python library: https://github.com/trail-ml/agent-cow-python
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。