arXiv:2606.29280cs.LGcs.AI2026-06

发现大模型教育劝导存在盲目干预缺陷,提出零泄漏决策流水线解决此问题。

Deterministic Decisions for High-Stakes AI. A Zero-Egress Pipeline with the Deployability of RAG and the Accuracy of Machine Learning

  • 用监督策略学习替代零样本提示,基于轨迹数据训练决策模型
  • 在800名学生数据上实现99.9%干预决策准确率,误报率降至43个百分点以下
  • 适合需要高可靠性决策的教育、医疗等高风险场景使用

我们识别出零样本大语言模型(LLM)教育劝导代理中一种此前未量化的失败模式:在无任务特定训练的情况下,它们会在后见之明最优策略建议不作为时仍推荐行动。在包含800名学生的开放大学学习分析数据集(OULAD)六臂消融实验中,第56天时,当最优参考策略认定70.1%的学生无需干预,零样本GPT-4o却仍推荐干预73%,导致43个百分点的假阳性率。商业RAG与SQL增强检索同样严重校准偏差;在10,000名学生规模下,每轮约产生4,300次不必要的顾问联系。监督策略学习可消除该偏差:基于同一参考标签轨迹训练的轨迹条件化ONNX决策变压器(DT)和快照XGBoost分类器,在严格前缀特征约束下均实现接近零的校准误差。DT在全部五类动作上达到宏F1 0.79(宏召回0.85),准确预测稀有减负动作,无动作翻转现象,决策延迟低于5毫秒CPU时间。两者性能相当;最终截止点上DT略胜于XGBoost,但差异不具统计显著性。研究范围限于受控奥拉克输入下的二级决策阶段(从EAV状态向量到监督策略);高保真度反映特征与参考对齐,非泛化高风险AI能力。最稳健发现为干预偏差对比,而非绝对准确率。此外揭示评估缺口:LLM作为评判者(DeepEval G-Eval)无法察觉干预偏差,反而奖励冗余劝导而非决策质量。

原文摘要 · Abstract (English)

We identify intervention bias as a previously unquantified failure mode of zero-shot large-language-model (LLM) educational advisory agents: without task-specific training, they recommend action when a hindsight-optimal oracle policy mandates inaction. In a six-arm ablation on the Open University Learning Analytics Dataset (N=800 students, four temporal cutoffs), at day 56 -- when the oracle designates 70.1% of students as needing no intervention -- zero-shot GPT-4o recommends action for 73%, a 43 percentage-point false-positive rate. Commercial RAG and SQL-augmented retrieval are comparably miscalibrated; at 10,000 students this implies about 4,300 unnecessary advisor contacts per cycle. Supervised policy learning eliminates this bias: a trajectory-conditioned ONNX Decision Transformer (DT) and a snapshot XGBoost classifier, trained on the same oracle-labelled trajectories under strict prefix-only features, both achieve near-zero calibration error. The DT reaches macro-F1 0.79 (macro-recall 0.85) across all five action classes, predicting even the rare load-reduction action without collapsing, at a 0% action flip rate and sub-5 ms CPU decision latency. The two supervised arms are on par; the DT's edge over XGBoost at the final cutoff is indicative only (unpaired across cohorts). Scope: we validate Stage-2 decision-making (EAV state vector to supervised policy) under controlled oracle input from structured OULAD data; high fidelity reflects feature-oracle alignment, not general high-stakes-AI capability. The most robust finding is the intervention-bias contrast, not the absolute accuracies. We also show an Evaluation Gap: LLM-as-judge scoring (DeepEval G-Eval) is blind to intervention bias, rewarding fluent over-prescription rather than decision quality.

高风险AI决策系统干预偏差监督学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。