智能决策中,如何精准调度人力纠错?
When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions
- 将AI评估视为信号,按需选择性获取人类判断以降低成本
- 在同行评审中仅用20-30%人力,达到接近全人工审核效果
- 线性模型下规则简洁可解释,计算成本低且性能不降
AI系统越来越多地辅助人类决策,提供复杂输入的初步评估。然而,这些评估常存在噪声或系统性偏差,关键问题在于:如何高效分配高成本的人力资源,在最关键环节修正AI输出?本文提出一种基于决策理论的通用框架,将AI评估视为因子级信号,人类判断作为可选的高成本信息源。针对最优选择问题,通过最大化与候选因子子集相关的奖励来设计策略。在非参数和线性模型下开发了估计方法,涵盖上下文与非上下文选择规则。在线性情形下,最优规则具有闭式表达,可清晰解释为因子重要性与残差方差的函数。应用于AI辅助同行评审时,该方法显著优于仅使用大语言模型的预测,性能接近全人工评审,同时仅需20%-30%的人力信息。不同规则下,基于线性模型的简化规则能大幅降低计算开销,且不影响最终预测表现。结果凸显了人类干预的价值与合理调度的效率。
原文摘要 · Abstract (English)
AI systems increasingly assist human decision making by producing preliminary assessments of complex inputs. However, such AI-generated assessments can often be noisy or systematically biased, raising a central question: how should costly human effort be allocated to correct AI outputs where it matters the most for the final decision? We propose a general decision-theoretic framework for human-AI collaboration in which AI assessments are treated as factor-level signals and human judgments as costly information that can be selectively acquired. We consider cases where the optimal selection problem reduces to maximizing a reward associated with each candidate subset of factors, and turn policy design into reward estimation. We develop estimation procedures under both nonparametric and linear models, covering contextual and non-contextual selection rules. In the linear setting, the optimal rule admits a closed-form expression with a clear interpretation in terms of factor importance and residual variance. We apply our framework to AI-assisted peer review. Our approach substantially outperforms LLM-only predictions and achieves performance comparable to full human review while using only 20-30% of the human information. Across different selection rules, we find that simpler rules derived under linear models can significantly reduce computational cost without harming final prediction performance. Our results highlight both the value of human intervention and the efficiency of principled dispatching.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。