一键测评与控制大模型智能体，精准衡量其工具使用与决策能力。
BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
- 构建13个工具调用智能体的基准测试，覆盖7类任务共2057个可验证任务。
- 通过真实数据库、图计算和闭式函数评估，确保答案可信且反映模型自主判断。
- 提供可视化平台实时监控智能体行为，支持远程终止与性能分析，适合部署者使用。
大型语言模型智能体可在多步骤中自主推理、调用工具并行动，但其核心能力——正确排序工具、在依赖关系下规划、判断不可信输入、生成内容的可靠性——难以准确衡量，仅靠排行榜无法满足需求。本文提出BekchiAI，包含一个用于测量智能体技能的基准测试和一个用于观察与控制实时运行智能体的平台。BekchiAI-Benchmark由13个基于ReAct的工具调用智能体组成，涵盖7个任务类别（算术、结构化/SQL、安全检测、URL溯源、规划、编排与工具策略），共2057个确定性、可执行的测试任务。每个任务的答案均通过运行标准SQL查询真实数据库、计算有向无环图（DAG）的精确调度或求解闭式λ表达式获得；其中还包含对抗性安全样本与故意不完善的签名扫描器，使评分反映模型自身判断，而非复制外部权威。除准确率外，定义了若干行为度量指标：工具调用遵从性、URL幻觉与来源匹配度、每模型的令牌成本。对四款模型（Qwen3.7-Max、gemma-4-31B-it、gemma4:26b、gpt-oss-120b）的对比显示，家族内差异显著，而整体平均值意义有限。所有基准测试均通过提供脚本执行。BekchiAI-Platform是一个配套的基于Web的可观测性与控制层，支持完整令牌与延迟追踪，以及远程运行终止。基准测试、评估工具与平台已公开发布。
原文摘要 · Abstract (English)
Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,057 deterministic, committed test tasks. Every task is verifier-checkable gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model's own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy-tool-call adherence, URL hallucination and source-match, and per-model token cost and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。