提出可审计计算轨迹标准，解决大模型数据代理答案正确但推理无效的问题。
Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems
- 用执行合约绑定用户意图与可执行查询，确保推理过程可验证
- 实测显示答案准确率高但计算轨迹有效率仅40%左右
- 适合关注系统可靠性、需可审计推理的工业级大模型应用
大模型数据代理的答案准确性不足以反映其可靠性。在结构化数据任务中，一个基准正确的答案可能由无效的计算轨迹生成。本文提出「轨迹完整性」，作为部署可靠性评估标准，要求计算过程具备显式性、可执行性、模式有效性、操作符忠实性、可重放性、答案一致性及可审计性。识别出「结构差距」是导致该问题的根本原因：自然语言推理和自由格式解释无法可靠生成真实系统所需的底层操作程序。通过执行合约（execution contracts）实现轨迹完整性，其内容包括用户意图、模式元素、操作计划、假设、可执行查询、验证状态和最终答案关联。引入CAIT率（正确答案/无效轨迹率），衡量仅以答案为准的评估将无计算支持输出误判为成功的情况。在BIRD Mini-Dev测试中，直接SQL、操作摘要+SQL、合同优先SQL的答案准确率分别为20%、22%、24%，而轨迹完整性通过率分别为39%、43%、40%，CAIT率仍高达55%、59.1%、45.8%，表明答案准确率、轨迹有效性与隐性失败风险是三个独立评估信号。因此，真实世界的大模型数据代理应不仅考察输出是否匹配参考答案，更需验证其背后的计算是否可审计。
原文摘要 · Abstract (English)
Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integrity, a deployment reliability criterion for evaluating whether the computation recorded behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We identify the Structure Gap as the deployment failure mode that makes Trace Integrity necessary: natural-language reasoning and free-form rationales do not reliably specify the operator-level programs required by real-world systems. We operationalize Trace Integrity with execution contracts, structured artifacts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. We also introduce CAIT (Correct Answer / Invalid Trace) Rate, which measures how often answer-only evaluation counts computationally unsupported outputs as successes. In an empirical demonstration on BIRD Mini-Dev, Direct SQL, Operation Summary + SQL, and Contract-First SQL achieve answer accuracies of 20%, 22%, and 24%, while their Trace Integrity Pass Rates are 39%, 43%, and 40% and their CAIT Rates remain high at 55%, 59.1%, and 45.8%, showing that answer accuracy, trace validity, and silent-failure risk are distinct evaluation signals. Real-world LLM data agents should, therefore, be evaluated not only by whether their outputs match a reference answer, but by whether those outputs are backed by auditable computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。