让专家经验变成可迭代优化的智能规则，自动提升合规决策准确率。
Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents
- 用错误分析驱动规则迭代，逐步修补缺失、错误和冲突问题。
- 8轮优化后规则准确率达79.6%，远超一次蒸馏的70%。
- 生成的Python代码比LLM提示词高9.8个百分点，适合生产部署。
企业专家在审计、合规和合同审查中隐含的决策规则，可通过迭代错误分析系统性恢复并改进。我们提出Trace2Policy，其核心机制EISR（误差驱动的迭代技能精炼）以可读规则文档为优化目标：每轮在验证集上执行规则，按根本原因聚类错误为缺失、错误或冲突三类，应用针对性修复，并仅提交通过回归测试的变更。针对这类敏感合规、低基率决策任务，我们发现规则质量是性能主导因素：在五种大模型上，单次蒸馏在部署数据集上稳定在约70%，而经过八轮EISR优化后，规则编译为确定性Python代码时准确率达79.6%，推理阶段零调用大模型。执行形式显著提升性能：同一优化内容在生产环境中以编译代码形式运行，比作为LLM提示词高出9.8个百分点，该形式与工程方案经22天部署磨合成熟。在某大型物流商部署22天（3,349个审计案例），编译流水线优于原纯大模型基线（72.7%）；在这些校准过的低基率任务上，重新启用大模型回退反而单调降低准确率。一个基于大模型的变体Auto-EISR，每轮成本仅5-10美元，相比约70人时的专家工作量，且无需重工程即可迁移至四个公开基准，涵盖法律推理（LegalBench）和流程挖掘决策（BPIC 2012）。
原文摘要 · Abstract (English)
Decision rules that enterprise experts apply tacitly -- in auditing, compliance, and contract review -- can be systematically recovered and improved through iterative error analysis. We present \textbf{Trace2Policy}, whose core mechanism -- \textbf{EISR} (\textbf{E}rror-driven \textbf{I}terative \textbf{S}kill \textbf{R}efinement) -- maintains a human-readable rule document as its optimization target: each round executes the rules on a validation set, clusters errors by root cause into MISSING, WRONG, or CONFLICT types, applies targeted patches, and commits only those that pass a regression gate. \textbf{For this class of compliance-sensitive, skewed-base-rate decision tasks, we identify rule quality -- not model capability -- as the dominant performance lever}: across five LLMs, one-shot distillation plateaus near $\sim$70\% on the deployed pool, while eight EISR rounds lift the same rules to 79.6\% when compiled into deterministic Python -- zero LLM calls at inference. \textbf{Execution form compounds the gain: in production, the same EISR-refined content runs 9.8~pp higher as compiled Python than as an LLM prompt, a form-and-engineering bundle the 22-day deployment matured together.} Deployed for 22 days at a major logistics carrier (3,349 audit cases), the compiled pipeline outperforms the pure-LLM baseline it replaced (72.7\%); on these calibrated, skewed-base-rate workloads, re-enabling LLM fallback monotonically degrades accuracy. An LLM-driven variant, \textbf{Auto-EISR}, reproduces this refinement at \$5--\$10 per cycle versus $\sim$70 expert-hours, and transfers to four public benchmarks spanning legal reasoning (LegalBench) and process-mining decisions (BPIC 2012) without re-engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。