现有评测无法反映大模型在真实生产环境中的表现，新框架RAMP可实时评估其长期任务执行能力。
Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
- 构建基于YatCC的运行时评估框架RAMP，模拟真实软件工程流程
- 15个主流模型在长流程中任务完成率从100%降至20%，无一完整通过
- 揭示失败传播与资源浪费问题，适合关注模型落地可靠性的开发者
大语言模型代理正从编程助手演变为自主软件工程系统。然而，现有评估方法仍集中于静态、孤立、短周期基准，难以捕捉真实生产工作流的动态复杂性。为此，本文提出RAMP，一个基于YatCC集成平台的生产级运行时评估框架。RAMP通过标准化编排与执行接口，引入具有串行依赖和复杂工具链交互的编译器构建负载，并设计分阶段恢复机制，分析部分失败下的执行行为。框架融合以效用为导向的多维指标，综合评估结果质量与过程效率。对15个主流模型的运行时评估显示，任务完成率在串联流程中从初始阶段的100%逐步下降至最终阶段的20%，无一模型能成功完成全流程。运行时分析揭示系统性失败传播与显著资源低效，不同模型间计算成本差异达三个数量级。结果表明，RAMP推动了代理模型评估向持续、可观测、生产导向的方向发展。
原文摘要 · Abstract (English)
LLM agents are rapidly evolving from coding assistants into autonomous software engineering systems. However, existing evaluation methodologies remain largely centered on static, isolated, and short-horizon benchmarks that fail to capture the dynamic complexity of real-world production workflows. As a result, benchmark performance may poorly reflect practical capability under realistic runtime environments involving long execution chains, tool interactions, dependency management, and iterative feedback loops. We thus present RAMP, a production-grounded infrastructure for assessing long-horizon software engineering agents. Built upon the YatCC integrated platform, RAMP provides a unified runtime assessment architecture through standardized orchestration and execution interfaces. RAMP introduces realistic compiler-construction workloads with serial dependencies and complex toolchain interactions, together with a staged recovery mechanism for analyzing execution behavior under partial workflow failure. The framework further incorporates utility-oriented multi-dimensional metrics that jointly evaluate outcome quality and process efficiency. We conduct runtime assessments across 15 mainstream models and observe substantial capability degradation that remains largely invisible to conventional isolated benchmarks. Task completion rates progressively collapse across serial workflows, dropping from 100% in the initial stage to only 20% in the final stage, while none of the evaluated models successfully completes the entire pipeline. Runtime analysis reveals systematic failure propagation and significant resource inefficiencies, with computational costs differing by up to three orders of magnitude among comparable models. These findings suggest RAMP advances agentic model evaluation toward continuous, runtime-observable, and production-grounded assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。