提出对抗性评估框架,检验规划智能体在对手攻击下的表现稳定性。
AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents
- 用对抗响应集测试规划生成器的鲁棒性,模拟真实竞争环境。
- 单次响应时蓝方优势达0.518,八次候选响应降至0.486,胜率从0.900降为0.820。
- 支持多智能体协作与可追溯的规划修正轨迹分析,适合研究对抗评估方法。
结构化规划代理常被孤立评估其计划质量,但许多现实规划任务需考察候选方案在对手搜索回应时的表现。本文提出AdvPlan-Bench,一个离线对抗评估基准。该基准包含类型化计划、对抗响应集、选择诊断与可追溯的候选前沿指标。计划以带可选分支的类型化动作链表示,赋予合成质量分;通过蓝方对红方的优势和纳什差距诊断进行对比,并使用透明启发式评分标准评估约束一致性。在涵盖五种规划模板的150个合成场景中,采样最佳响应策略(生成8个候选)使蓝方优势从0.518降至0.486,胜率从0.900降至0.820,相比单样本响应。离线大模型策略基线取得0.496的蓝方优势与0.700的胜率,两阶段多智能体理事会达到0.509的蓝方优势与0.813的胜率。六百条评分记录的三评者敏感性研究显示0.978的组内一致性。AdvPlan-Bench并非实际规划工具,不提供真实决策质量证据,而是可复现的基准工具,用于研究对抗评估、响应预算敏感性、候选前沿及多智能体批判与修订过程。
原文摘要 · Abstract (English)
Structured plan-generation agents are often evaluated as if a plan has quality in isolation, yet many realistic planning tasks require asking how a candidate behaves when another agent can search for responses. We introduce AdvPlan-Bench, an offline benchmark for adversarial evaluation of structured plan-generation agents. The contribution is a general evaluation object: a typed plan, an adversarial response set, selector diagnostics, and traceable candidate-frontier metrics. AdvPlan-Bench represents plans as typed action chains with optional branches, assigns synthetic quality scores, compares opposing plans with BLUE-vs-RED advantage and Nash-gap diagnostics, and evaluates qualitative constraint coherence with a transparent heuristic rubric. In 150 synthetic scenarios spanning five planning templates, a sampled best-response policy that draws eight response candidates reduces BLUE advantage from .518 to .486 and BLUE win rate from .900 to .820 relative to a single-sample response. An offline LLM-policy contract baseline reaches .496 BLUE advantage and .700 BLUE win rate, while a two-stage multi-agent council obtains .509 BLUE advantage and .813 BLUE win rate. A three-rater rubric-sensitivity study over 600 rating records yields .978 inter-rater agreement. AdvPlan-Bench is not an operational planner and provides no evidence about real-world decision quality; it is a reproducible benchmark artifact for studying adversarial plan evaluation, response-budget sensitivity, candidate frontiers, and multi-agent critique-and-revision traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。