用运筹学构建可信的生成式智能系统,确保自主决策安全可靠。
Assured autonomy: How operations research powers and orchestrates generative AI systems
- 基于流模型将生成过程视为可审计的确定性传输过程。
- 通过对抗鲁棒性评估,增强系统在极端情况下的稳定性与容错能力。
- 适合关注安全关键领域自主系统设计的研究者与工程师。
生成式人工智能正从对话助手转向具有感知、决策和行动能力的代理系统。这一转变带来自治悖论:随着系统自主性提升,应具备更严格的结构、明确的约束和更强的尾部风险管控。我们指出,若不与可验证可行性、抗分布偏移及高后果场景压力测试机制结合,随机生成模型在操作环境中可能脆弱。为此,提出一个基于运筹学的可信自治概念框架,包含两个互补方法:其一,流基生成模型将生成建模为由常微分方程描述的确定性传输,实现可审计性、约束感知生成,并关联最优传输、鲁棒优化与序贯决策控制;其二,从对抗鲁棒性视角定义操作安全,对不确定性或模糊集内的最坏扰动作决策规则评估,使未建模风险成为设计的一部分。该框架明确了运筹学角色从求解器演变为守门人乃至系统架构师,涵盖控制逻辑、激励协议、监控机制与安全边界。这为安全关键与可靠性敏感场景中的可信自治研究指明方向。
原文摘要 · Abstract (English)
Generative artificial intelligence (GenAI) is shifting from conversational assistants toward agentic systems -- autonomous decision-making systems that sense, decide, and act within operational workflows. This shift creates an autonomy paradox: as GenAI systems are granted greater operational autonomy, they should, by design, embody more formal structure, more explicit constraints, and stronger tail-risk discipline. We argue that stochastic generative models can be fragile in operational domains unless paired with mechanisms that provide verifiable feasibility, robustness to distribution shift, and stress testing under high-consequence scenarios. To address this challenge, we develop a conceptual framework for assured autonomy grounded in operations research (OR), built on two complementary approaches. First, flow-based generative models frame generation as deterministic transport characterized by an ordinary differential equation, enabling auditability, constraint-aware generation, and connections to optimal transport, robust optimization, and sequential decision control. Second, operational safety is formulated through an adversarial robustness lens: decision rules are evaluated against worst-case perturbations within uncertainty or ambiguity sets, making unmodeled risks part of the design. This framework clarifies how increasing autonomy shifts OR's role from solver to guardrail to system architect, with responsibility for control logic, incentive protocols, monitoring regimes, and safety boundaries. These elements define a research agenda for assured autonomy in safety-critical, reliability-sensitive operational domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。