Qiushi引擎在硬核科研任务上实现10%全任务完成率，表现远超官方平均。
Qiushi Engine on AstaBench E2E-Bench-Hard
- 采用DeepSeek-v4pro-preview模型，自主完成从实验设计到报告交付的全流程。
- 40项任务中完成4项，满足416个评分条目，准确率达82.1%。
- 适合关注自动化科研代理的开发者与研究者，尤其关注系统可靠性与成本效益。
本报告分析了Qiushi Engine v0.8在AstaBench E2E-Bench-Hard全部40个测试任务上的表现，该基准要求智能体自主完成研究问题的实验设计、代码实现、实际执行、结果分析及报告生成全过程。评估选用DeepSeek-v4pro-preview作为模型后端。官方AstaBench排行榜显示平均得分为0.816，每任务平均成本为15.209美元；而全精度本地重计算成本为81.59±1.87美元。共有4项任务完全满足所有评分标准，全任务完成率为4/40=10%，比官方最佳代理约3%高出7个百分点，约为其3.3倍。在总计507个评分条目中，共满足416项（82.1%）。官方评分记录与40份Meta-Trace数据表明系统持续生成并验证报告、代码与实验成果；主要短板在于重复运行、外部依赖、指定指标和消融实验的缺失。报告详述了基准设定、系统流程、汇总结果、代表性案例及解释局限性。
原文摘要 · Abstract (English)
This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a research question through experimental design, code implementation, actual execution, result analysis, and report delivery. Qiushi Engine is model-configurable; this evaluation selected DeepSeek deepseek-v4pro-preview as the model backend. The official AstaBench leaderboard records a score of 0.816 and an average benchmark cost of USD 15.209 per task, while the full-precision local recomputation is $81.59 \pm 1.87$. Four tasks satisfied every rubric item, yielding a full-task completion rate of 4/40 = 10% -- 7 percentage points above, and about 3.3 times, the approximately 3% best rate reported for AstaBench's official agents. Across 507 required rubric items, 416 were satisfied (82.1%). Official scoring archives and 40 Meta-Trace records show sustained production and verification of reports, code, and experimental artifacts; the principal gaps lie in repeated runs, external dependencies, specified metrics, and ablation studies. The report explains the benchmark, system workflow, aggregate results, representative cases, and limits of interpretation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。