让AI在不确定时主动说‘待定’,并可审计决策过程。
Auditable Decision Models with Learned Abstention and Real-Time Steering

- 模型输出仅限是/否/待定三类,待定为学习得到的拒答选项
- 测试集上准确率82.6%,各类别F1在79.6%至84.9%之间
- 支持运行时调控与审计,适合高风险决策系统使用
生产环境中的AI系统常面临不完整、矛盾或证据不足的情况。强制分类器会强行给出标签,而生成式系统则可能产生难以审计的输出。本文提出EvaluatorDPT,一种有限决策控制模型,输出YES、NO或TBD,其中TBD为通过学习获得的拒答结果,而非事后置信度规则。模型采用Transformer编码器,包含主决策头及用于价值和情感/情绪的结构化辅助通道。接口形式领域无关:部署域提供证据与政策阈值,模型输出可由记录的操作阈值和验证后的语义信号在推理时控制。在保留测试集(n=44,597)上的评估显示,模型准确率为0.8260,宏平均F1为0.8252,各类别F1分别为0.8314(YES)、0.8486(NO)、0.7956(TBD)。评估还包含校准证据(验证集ECE=0.0338)、阈值扫描结果、多种子稳定性检查、混淆矩阵及可复现命令。
原文摘要 · Abstract (English)
Production AI systems often operate with incomplete, conflicting, or insufficient evidence. Forced classifiers collapse such cases into action labels, while generative systems can produce outputs that are difficult to interpret as auditable execution decisions. We study operational decision control for AI systems, where uncertainty must be explicitly routable, policy-governed, and auditable rather than hidden inside forced predictions or free-form generation. We present EvaluatorDPT, a bounded decision-control model that predicts YES, NO, or TBD, where TBD is learned as a deferral outcome rather than added only as a post-hoc confidence rule. The model uses a transformer encoder with a primary bounded-decision head and structured auxiliary channels for values and emotions/sentiments. The interface is domain-agnostic in form: a deployment domain supplies evidence and policy thresholds, while the model emits a bounded distribution that can be controlled at inference time through recorded operating thresholds and, when validated, auxiliary semantic signals. For the evaluated model version, we report decision performance on held-out validation and test splits; auxiliary emotion metrics are omitted because the emotion head is disabled for this evaluation. On the held-out test split (n=44,597), the model achieves Accuracy = 0.8260 and Macro F1 = 0.8252, with per-class F1 of 0.8314 (YES), 0.8486 (NO), and 0.7956 (TBD). The evaluation record also includes calibration evidence (ECE = 0.0338 on validation), threshold-sweep outputs, multi-seed stability checks, confusion matrices, and reproducibility commands. Our main contribution is a bounded execution interface in which deferral is learned, inference-time routing remains inspectable, auxiliary signals provide a path to auditable behavior control, and evaluation evidence supports external review.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。