评测大模型代理在真实压力下的可靠性，发现接口限流最致命。
ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions
- 构建三维度可靠性评测框架：重复执行一致性、任务扰动鲁棒性、工具故障容错性
- 实测扰动使成功率从96.9%降至88.1%，限流是破坏力最强的故障类型
- 适用于评估生产级智能体，尤其关注成本与性能平衡的开发者
现有工具调用型大模型代理基准主要报告单次运行成功率，忽略生产环境所需的可靠性特性。本文提出 extbf{ReliabilityBench}，一个从三个维度评估代理可靠性的基准：(i) 使用$ ext{pass}^k$衡量重复执行的一致性，(ii) 在强度$ε$下对语义等价任务扰动的鲁棒性，(iii) 在强度$λ$下受控工具/API故障的容错能力。该基准引入统一的可靠性曲面$R(k,ε,λ)$，定义正确性的动作元关系（基于终态等价而非文本相似），并设计类混沌工程的故障注入框架（超时、限流、部分响应、模式漂移）。我们在四个领域（日程安排、旅行、客户支持、电商）对两个模型（Gemini 2.0 Flash、GPT-4o）和两种架构（ReAct、Reflexion）进行了1,280次实验。仅扰动即导致成功率从$ε=0$时的96.9%下降至$ε=0.2$时的88.1%。消融实验表明限流是最具破坏性的故障。在综合压力下，ReAct比Reflexion更鲁棒，且Gemini 2.0 Flash在远低于GPT-4o的成本下达到相当的可靠性。ReliabilityBench为评估大模型代理生产就绪状态提供了系统化框架。
原文摘要 · Abstract (English)
Existing benchmarks for tool-using LLM agents primarily report single-run success rates and miss reliability properties required in production. We introduce \textbf{ReliabilityBench}, a benchmark for evaluating agent reliability across three dimensions: (i) consistency under repeated execution using $\mathrm{pass}^k$, (ii) robustness to semantically equivalent task perturbations at intensity $ε$, and (iii) fault tolerance under controlled tool/API failures at intensity $λ$. ReliabilityBench contributes a unified reliability surface $R(k,ε,λ)$, \textit{action metamorphic relations} that define correctness via end-state equivalence rather than text similarity, and a chaos-engineering-style fault injection framework (timeouts, rate limits, partial responses, schema drift). We evaluate two models (Gemini 2.0 Flash, GPT-4o) and two agent architectures (ReAct, Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes. Perturbations alone reduce success from 96.9% at $ε=0$ to 88.1% at $ε=0.2$. Rate limiting is the most damaging fault in ablations. ReAct is more robust than Reflexion under combined stress, and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at much lower cost. ReliabilityBench provides a systematic framework for assessing production readiness of LLM agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。