AI自主设计新型神经网络架构，性能超越人类设计。
Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design
- 用11个智能体探索计算原语，生成多种新架构。
- 新模型在多个任务上比Llama 3.2高2.4%~3.8%，训练更快。
- 适合研究自动生成模型与高效架构优化的团队。
为实现递归自我改进，我们研究大语言模型代理自主设计超越标准Transformer的基础模型。提出双框架：AIRA-Compose用于高层架构搜索，AIRA-Design用于底层机制实现。AIRA-Compose使用11个代理在24小时内探索基础计算原语，评估数百万参数候选，将最优设计扩展至350M、1B和3B规模。生成14种架构，分为两类：AIRAformers（基于Transformer）和AIRAhybrids（Transformer-Mamba）。在1B规模预训练后，这些模型持续优于Llama 3.2和Composer发现的基线。在下游任务中，AIRAformer-D和AIRAhybrid-D分别比Llama 3.2提升2.4%和3.8%准确率。此外，AIRA-Compose发现具有高效率扩展前沿的模型：AIRAformer-C比Llama 3.2快54%、比Composer最佳Transformer快71%；AIRAhybrid-C比Nemotron-2快23%，比Composer最佳混合模型快37%。AIRA-Design部署20个代理，编写新型注意力机制以处理长程依赖，并生成高性能训练脚本。在Long Range Arena基准上，代理设计架构在文档匹配和文本分类任务上分别仅落后人类最先进水平2.3%和2.6%。在Autoresearch基准上，Greedy Opus 4.5在固定时间预算下达到0.968验证比特/字节，超过已发表最低值。整体表明，AI代理可自主发现媲美或超越人工设计的架构与算法优化，建立下一代基础模型发现的新范式，迈向递归自我改进的关键一步。
原文摘要 · Abstract (English)
Toward recursive self-improvement, we investigate LLM agents autonomously designing foundation models beyond standard Transformers. We introduce a dual-framework approach: AIRA-Compose for high-level architecture search, and AIRA-Design for low-level mechanistic implementation. AIRA-Compose uses 11 agents to explore fundamental computational primitives under a 24-hour budget. Agents evaluate million-parameter candidates, extrapolating top designs to 350M, 1B, and 3B scales. This yields 14 architectures across two families: AIRAformers (Transformer-based) and AIRAhybrids (Transformer-Mamba). Pre-trained at 1B scale, these consistently outperform Llama 3.2 and Composer-found baselines. On downstream tasks, AIRAformer-D and AIRAhybrid-D improve accuracy by 2.4% and 3.8% over Llama 3.2. Furthermore, AIRA-Compose finds models with highly efficient scaling frontiers: AIRAformer-C scales 54% and 71% faster than Llama 3.2 and Composer's best Transformer, while AIRAhybrid-C outscales Nemotron-2 by 23% and Composer's best hybrid by 37%. AIRA-Design tasks 20 agents with writing novel attention mechanisms for long-range dependencies and high-performing training scripts. On the Long Range Arena benchmark, agent-designed architectures reach within 2.3% and 2.6% of human state-of-the-art on document matching and text classification. On the Autoresearch benchmark, Greedy Opus 4.5 achieves 0.968 validation bits-per-byte under a fixed time budget, surpassing the published minimum. Together, these frameworks show AI agents can autonomously discover architectures and algorithmic optimizations matching or surpassing hand-designed baselines. This establishes a powerful paradigm for discovering next-generation foundation models, marking a clear step toward recursive self-improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。