用贝叶斯校准代替单元测试，发现并修复大模型写的概率程序错误。
Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models
- 以贝叶斯工作流为验证器，通过后验预测检查等方法识别统计错误。
- 在14类错误中检测准确率达AUC 0.97，参考程序下误报率仅2%。
- 适合需高可信概率建模的科研与工程场景，尤其对抗性调试需求强的项目。
语言模型越来越多地编写概率程序（如NumPyro、Stan或Pyro），但一个能编译、运行并通过所有单元测试的程序仍可能在统计上错误——例如对重尾数据使用高斯似然、对过度离散计数使用泊松分布、无效先验支持或病态参数化。正确的验证工具不应是测试套件，而应是贝叶斯工作流本身：后验预测检查、基于模拟的校准、采样器诊断（$ ilde{R}$、发散、有效样本量）及保留预测密度。我们沿三个维度研究这一校准基准。检测方面，在涵盖10个模型族的200个实例中，针对14类错设类型，该方法在给定正确参考程序时达到AUC 0.97（FPR为2%时准确率88%，为理论上限）；完全无参考的版本也达62%–78%（来自小规模自动模型搜索），而单元测试验证器仅为0%。修复方面，在15个模型上使用校准反馈进行大模型修复循环，显著优于单元测试反馈——后者甚至比无反馈更差，因通过测试会引发虚假信心，抑制修复；在强但未饱和的模型中，校准反馈显著提升（GPT-5.1从33%→92%，Claude从75%→100%；配对麦内玛检验，n=228）。现实情况中，大模型根据中性简述自行编写的可运行程序，15%–47%存在统计错设（单元测试全未捕捉），而校准引导修复显著优于大模型自评、贝叶斯流程清单和数据摘要自检。三者结论一致：对概率程序而言，正确性在于校准，而非编译。
原文摘要 · Abstract (English)
Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization. The right verifier is therefore not a test suite but the Bayesian workflow itself: posterior predictive checks, simulation-based calibration, sampler diagnostics ($\hat R$, divergences, ESS), and held-out predictive density. We study this calibration oracle along three axes. \textbf{Detection:} on a benchmark of $14$ misspecification types across $10$ model families ($200$ instances), it flags the bug with AUC $0.97$ ($88\%$ at $2\%$ FPR \emph{when handed the correct reference program, an upper bound}) -- and a fully \emph{reference-free} version that uses no correct program reaches $62$--$78\%$ (the upper figure from a small automated model search), versus $0\%$ for a unit-test oracle. \textbf{Repair:} used as feedback in an LLM repair loop across fifteen models, calibration significantly outperforms unit-test feedback -- which is itself \emph{significantly worse than no feedback at all}, a passing test inducing false confidence that suppresses repair -- and improves over no feedback on strong-but-unsaturated models (GPT-5.1 $33{\to}92\%$, Claude $75{\to}100\%$; paired McNemar, $n{=}228$). \textbf{Reality:} on programs LLMs write from scratch for neutral briefs, $15$--$47\%$ of runnable ones are statistically misspecified (unit tests catch none), and calibration-guided repair significantly beats LLM-as-judge review, a Bayesian-workflow checklist, and data-summary self-debug. Across all three, the lesson is the same: for probabilistic programs, correctness is calibration, not compilation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。