首次大规模实测22个智能体框架，发现调度质量比推理能力更关键。
Agentic Frameworks for Reasoning Tasks: An Empirical Study
- 统一测试22个智能体框架，按架构分类并评估表现。
- 12个框架稳定运行，数学题准确率仅44.35%，远低于其他任务。
- 智能体选型需重视内存控制与成本管理，避免重复失败导致浪费。
近期智能体框架使AI能执行复杂推理与决策，但其推理性能、效率与实际适用性的对比证据仍有限。为填补这一空白，我们在三个推理基准（BBH、GSM8K、ARC）上对22个广泛使用的智能体框架进行了实证评估。这些框架从2023年1月至2025年7月收集的1200个GitHub仓库中筛选，并根据架构设计分类。所有框架在统一设置下评估，测量推理准确率、执行时间、计算成本及跨基准一致性。结果表明，22个框架中有19个完成全部三项基准测试。其中12个表现稳定，平均准确率74.6%-75.9%，每任务耗时4-6秒，成本0.14-0.18美分。表现较差主要源于调度问题而非推理能力限制：例如，Camel因上下文无限增长，在11天内未能完成BBH；Upsonic因重复提取失败触发高成本重试，单日消耗1434美元；AutoGen和Mastra则因迭代交互导致提示长度增加却未提升效果，耗尽API配额。数学推理表现显著下降：在GSM8K上平均准确率为44.35%，远低于BBH的89.80%和ARC的89.56%。本研究首次提供针对推理密集型软件工程任务的智能体框架大规模实证比较，表明框架选择应优先考虑调度质量，尤其是内存控制、错误处理与成本管理。
原文摘要 · Abstract (English)
Recent advances in agentic frameworks have enabled AI agents to perform complex reasoning and decision-making. However, evidence comparing their reasoning performance, efficiency, and practical suitability remains limited. To address this gap, we empirically evaluate 22 widely used agentic frameworks across three reasoning benchmarks: BBH, GSM8K, and ARC. The frameworks were selected from 1,200 GitHub repositories collected between January 2023 and July 2025 and organized into a taxonomy based on architectural design. We evaluated them under a unified setting, measuring reasoning accuracy, execution time, computational cost, and cross-benchmark consistency. Our results show that 19 of the 22 frameworks completed all three benchmarks. Among these, 12 showed stable performance, with mean accuracy of 74.6-75.9%, execution time of 4-6 seconds per task, and cost of 0.14-0.18 cents per task. Poorer results were mainly caused by orchestration problems rather than reasoning limits. For example, Camel failed to complete BBH after 11 days because of uncontrolled context growth, while Upsonic consumed USD 1,434 in one day because repeated extraction failures triggered costly retries. AutoGen and Mastra also exhausted API quotas through iterative interactions that increased prompt length without improving results. We also found a sharp drop in mathematical reasoning. Mean accuracy on GSM8K was 44.35%, compared with 89.80% on BBH and 89.56% on ARC. Overall, this study provides the first large-scale empirical comparison of agentic frameworks for reasoning-intensive software engineering tasks and shows that framework selection should prioritize orchestration quality, especially memory control, failure handling, and cost management.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。