模型生成策略路线后，用代码自持的预测评分，解决延迟真相问题。
From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth
- 用代码生成临时预测排名，不依赖即时验证
- 回顾性测试中预测准确率AUC达0.756，优于人工判断
- 适用于需延迟验证的决策场景，如投资或战略规划
许多模型输出评估依赖可在评估时验证的合约，或在运行周期内返回的反馈。本文研究了一种互补场景：真实结果延迟、被截断或保密，因此确定性代码无法在评分时检查正确性，必须改而发出由代码掌控的临时预测。RouteCast 在此框架下实现了模型生成的结构化战略路线评估：模型提出候选路线与结构因子；时点证据、参考类别与确定性变换生成临时预测排名；后续实际结果用于评估该预测。在21个二元结果案例（6正15负）的回溯试点中，完整包式RouteCast得分初步展现出区分能力（AUC 0.756，95%置信区间[0.471,0.980]），盲测LLM法官得AUC 0.678 [0.419,0.897]，身份暴露的LLM法官得AUC 0.761 [0.515,0.944]，符合识别或结果相关泄露风险。同一子集的预注册分解消融实验表明，将相同输入转为分阶段类型化路线与整体包评分无显著差异（Delta AUC = -0.144，95% CI [-0.471,0.176]），也与确定性启发法相当（Delta AUC = -0.089，95% CI [-0.412,0.278]）。试点证明了可审计可行性并揭示失败模式，但未建立前瞻性校准、因果决策改进、路线分解优势或跨领域有效性。
原文摘要 · Abstract (English)
Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast. RouteCast instantiates this regime for model-generated typed strategic routes: models propose candidate routes and structured factors; point-in-time evidence, reference classes, and deterministic transformations produce a provisional forecast-ranking; later outcomes evaluate the forecast. In a retrospective venture pilot on 21 binary-outcome cases (6 positive, 15 negative), the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756, 95% CI [0.471,0.980]), while a blind LLM judge reached AUC 0.678 [0.419,0.897] and an identity-exposed LLM judge reached AUC 0.761 [0.515,0.944], consistent with recognition- or outcome-related leakage risk. A preregistered decomposition ablation on the same binary subset found that converting the identical inputs into typed staged routes was indistinguishable from the whole-packet score (Delta AUC = -0.144, 95% CI [-0.471,0.176]) and from a deterministic heuristic (Delta AUC = -0.089, 95% CI [-0.412,0.278]). The pilot establishes an auditable feasibility result and exposes failure modes; it does not establish prospective calibration, causal decision improvement, route-decomposition advantage, or cross-domain validity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。