让大模型生成可执行代码,实现可验证的决策透明化。
From Stochastic Answers to Verifiable Reasoning: Interpretable Decision-Making with LLM-Generated Code
- 用大模型生成可运行的决策代码,避免逐条评估
- 在创始人筛选中达37.5%精准率,优于GPT-4o
- 结果可追溯、可审计,适合高风险决策场景
大型语言模型(LLMs)在高风险决策中应用日益广泛,但现有方法难以兼顾可扩展性、可解释性和可复现性。黑箱模型隐藏推理过程,而基于大模型的规则系统依赖逐样本评估,导致成本随数据量增长,并产生随机且易幻觉的输出。本文提出将大模型重构为代码生成器,而非逐实例评估者:单次调用生成可执行、人类可读的决策逻辑,对结构化数据进行确定性运算,消除逐样本调用,实现可复现与可审计预测。结合精度提升、二项式显著性检验和覆盖过滤进行自动化统计验证,并通过聚类差距分析迭代优化决策逻辑,无需人工标注。在创业公司创始人筛选任务中验证该框架,该任务为罕见事件预测,需强可解释性。在包含4,500名创始人的VCBench基准上,基础成功率为9%,本方法实现37.5%精确率与F0.5分数25.0%,优于GPT-4o(30.0%精确率,F0.5分数25.7%),同时保持完整可解释性。每个预测均可追溯至基于人类可读属性的可执行规则,实证了可验证、可解释的大模型决策机制。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used for high-stakes decision-making, yet existing approaches struggle to reconcile scalability, interpretability, and reproducibility. Black-box models obscure their reasoning, while recent LLM-based rule systems rely on per-sample evaluation, causing costs to scale with dataset size and introducing stochastic, hallucination-prone outputs. We propose reframing LLMs as code generators rather than per-instance evaluators. A single LLM call generates executable, human-readable decision logic that runs deterministically over structured data, eliminating per-sample LLM queries while enabling reproducible and auditable predictions. We combine code generation with automated statistical validation using precision lift, binomial significance testing, and coverage filtering, and apply cluster-based gap analysis to iteratively refine decision logic without human annotation. We instantiate this framework in venture capital founder screening, a rare-event prediction task with strong interpretability requirements. On VCBench, a benchmark of 4,500 founders with a 9% base success rate, our approach achieves 37.5% precision and an F0.5 score of 25.0%, outperforming GPT-4o (at 30.0% precision and an F0.5 score of 25.7%) while maintaining full interpretability. Each prediction traces to executable rules over human-readable attributes, demonstrating verifiable and interpretable LLM-based decision-making in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。