构建104个真实任务的智能体评测基准，提升测试可信度。
LiveAgentBench: Comprehensive Benchmarking of Agentic Systems Across 104 Real-World Challenges
- 基于社交媒体真实问题生成数据，确保任务贴近现实
- 涵盖374项任务，其中249项用于测试，验证模型实战能力
- 支持持续更新，适合评估智能体在真实场景的表现
随着大语言模型能力提升，通用人工智能代理在实际应用中日益普及。然而，现有评测基准存在显著局限，难以准确反映真实用户任务。为此，我们提出LiveAgentBench，一个包含104个真实场景的综合性评测基准，数据源自公开社交媒体提问及真实产品反馈。核心是自研的社交感知驱动数据生成（SPDG）方法，确保每道题具备现实相关性、任务复杂性和结果可验证性。我们用该基准评估多种模型、框架与商业产品，揭示其实际表现并识别改进空间。此次发布包含374项任务，其中125项用于验证，249项用于测试。SPDG机制支持持续从真实交互中引入新问题，保持评测前沿性。
原文摘要 · Abstract (English)
As large language models grow more capable, general AI agents have become increasingly prevalent in practical applications. However, existing benchmarks face significant limitations, failing to represent real-world user tasks accurately. To address this gap, we present LiveAgentBench, a comprehensive benchmark with 104 scenarios that reflect real user requirements. It is constructed from publicly sourced questions on social media and real-world products. Central to our approach is the Social Perception-Driven Data Generation (SPDG) method, a novel process we developed to ensure each question's real-world relevance, task complexity, and result verifiability. We evaluate various models, frameworks, and commercial products using LiveAgentBench, revealing their practical performance and identifying areas for improvement. This release includes 374 tasks, with 125 for validation and 249 for testing. The SPDG process enables continuous updates with fresh queries from real-world interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。