arXiv:2605.17046cs.LGcs.AI2026-05

首个单卡七任务基准,测试AI编程代理真实科研能力。

1GC-7RC: One Graphic Card -- Seven Research Challenges! How Good Are AI Agents at Doing Your Job?

  • 用单块显卡完成7类机器学习任务,仅允许改训练代码。
  • 各代理在7个任务中表现差异大,暴露规划与时间管理短板。
  • 开源可扩展,适合研究自主智能体的科研人员使用。

自主AI编程代理正成为工业界和学术界机器学习从业者的核心工具。然而,目前尚无标准化基准来评估其在多样领域中从零开始设计、实现和训练模型的能力。我们提出**1GC-7RC**(单显卡:七项研究挑战),包含语言建模、图像分类、语义分割、图学习、表格预测、时间序列预测和文本分类共七项任务。每项任务提供锁定的数据预处理与评估脚本,以及基线训练脚本;代理仅可修改训练代码,不可访问预训练权重(语义分割除外)、无网络连接,并需在40至120分钟内完成任务。我们评估了七款编码代理:五款专有模型(Claude Code with Sonnet 4.6、Opus 4.6、Opus 4.7;Codex CLI with GPT 5.5;OpenCode with Qwen 3.6+)和两款开源模型(OpenCode with Kimi K2.5、Kimi K2.6)。每个代理-任务组合运行5次,结果揭示了隐含的机器学习知识、规划能力和时间预算管理方面的显著差异。该基准、评测框架及全部实验资源已公开于GitHub:https://github.com/Strolchii/1GC-7RC-Benchmark,支持未来代理的可复现对比。因其模块化设计,该基准可拓展至新任务与领域,适配不同显卡预算,亦可用于多智能体研究,是自主研究代理的灵活平台。

原文摘要 · Abstract (English)

Autonomous AI coding agents are becoming a core tool for ML practitioners in industry and research alike. Despite this growing adoption, no standardized benchmark exists to evaluate their ability to design, implement, and train models from scratch across diverse domains. We introduce **1GC-7RC** (*Single Graphic Card: Seven Research Challenges*), a benchmark comprising seven ML tasks spanning language modeling, image classification, semantic segmentation, graph learning, tabular prediction, time-series forecasting, and text classification. Each task provides a locked data-preparation and evaluation script together with a baseline training script; the agent may only modify the training code, has no access to pretrained weights (with one controlled exception for semantic segmentation), no internet access, and must complete each task within a task-specific wall-clock budget (40-120 minutes) on a single GPU. We evaluate seven coding agents: five proprietary (Claude Code with Sonnet 4.6, Opus 4.6, and Opus 4.7; Codex CLI with GPT 5.5; and OpenCode with Qwen 3.6+) and two open-source (OpenCode with Kimi K2.5, Kimi K2.6). Across 5 runs per agent-task pair, we report substantial performance differences that reveal varying levels of implicit ML knowledge, planning ability, and time-budget management. The benchmark, harness, and all evaluation artifacts are publicly available on GitHub at https://github.com/Strolchii/1GC-7RC-Benchmark to facilitate reproducible comparison of future agents. Because our benchmark design is modular, the benchmark can be extended to new tasks and domains, adapted to different GPU budgets, and used to study multi-agent settings, making it a flexible platform for future research on autonomous research agents.

AI代理代码生成基准测试单卡训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。