让AI自己完成大模型评估，只需一句自然语言指令。
One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
- 用自然语言生成可执行的评估流程，自动规划基准测试。
- 支持自动获取数据集、统一格式并生成决策报告，减少人工干预。
- 适合需要高效、可追溯评估的大模型研发团队使用。
可靠评估对大语言模型的开发与部署至关重要，但实际操作中常需大量手动工作：研究者需选择合适基准、复现异构评估代码、配置数据集模式映射并解读聚合指标。为此，我们提出 One-Eval，一个智能评估系统，可将自然语言评估请求转化为可执行、可追溯且可定制的评估工作流。该系统集成三部分：(i) NL2Bench 用于意图结构化与个性化基准规划；(ii) BenchResolve 实现基准解析、自动数据集获取与模式标准化，确保可执行性；(iii) Metrics & Reporting 支持任务感知的指标选择与面向决策的报告输出，超越单一数值评分。系统还引入人机协同检查点，支持审查、编辑与回滚，并保留样本证据链以供调试与审计。实验表明，One-Eval 能从多样自然语言请求中端到端执行评估，极大降低用户负担，提升工业场景中的评估效率与可复现性。框架已开源：https://github.com/OpenDCAI/One-Eval。
原文摘要 · Abstract (English)
Reliable evaluation is essential for developing and deploying large language models, yet in practice it often requires substantial manual effort: practitioners must identify appropriate benchmarks, reproduce heterogeneous evaluation codebases, configure dataset schema mappings, and interpret aggregated metrics. To address these challenges, we present One-Eval, an agentic evaluation system that converts natural-language evaluation requests into executable, traceable, and customizable evaluation workflows. One-Eval integrates (i) NL2Bench for intent structuring and personalized benchmark planning, (ii) BenchResolve for benchmark resolution, automatic dataset acquisition, and schema normalization to ensure executability, and (iii) Metrics \& Reporting for task-aware metric selection and decision-oriented reporting beyond scalar scores. The system further incorporates human-in-the-loop checkpoints for review, editing, and rollback, while preserving sample evidence trails for debugging and auditability. Experiments show that One-Eval can execute end-to-end evaluations from diverse natural-language requests with minimal user effort, supporting more efficient and reproducible evaluation in industrial settings. Our framework is publicly available at https://github.com/OpenDCAI/One-Eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。