用搜索框架自动设计混合神经网络架构，性能超越Llama 3.2。
Composer: A Search Framework for Hybrid Neural Architecture Design
- 通过小规模搜索+可扩展策略，高效探索混合模型结构。
- 在350M-3B参数规模下，验证损失更低，下游任务准确率提升1.1%-8.3%。
- 适合追求高效高精度大模型的开发者与研究者使用。
将不同计算单元（如注意力、MLP）以不同比例组合的混合模型架构，在超越Transformer方面展现出潜力。已有研究指出，单元的交替顺序也会影响模型质量。然而，以往工作依赖人工设计，受限于庞大的设计空间和训练成本，难以发现适用于预训练的优质混合架构。本文提出一个模块化混合架构搜索框架Composer，先在小规模上探索模型结构，再通过提出的缩放策略外推至大规模。基于Composer，我们发现了新的混合大语言模型架构，其性能优于Llama 3.2。相比Llama 3.2及先前最优基线，新架构在350M-3B参数量级上持续降低验证损失，并在下游任务中实现最高达8.3%（平均1.1%-3.1%）的准确率提升，同时提升训练与推理效率。
原文摘要 · Abstract (English)
Hybrid model architectures that combine computational primitives (e.g., Attention, MLP) in different ratios have shown promising performance beyond Transformers. Some studies have shown that different interleavings of primitives can affect model quality as well. However, prior works explore the hybrid model architecture design space manually. Due to the large design space and training costs, discovering hybrid models that combine key computational primitives for pre-training is challenging. In this work, we take a principled approach in designing a modular hybrid model architecture search framework -- Composer. Composer explores model architectures at a small scale and extrapolates the top-performing model architectures to a larger scale using our proposed scaling strategies. Using Composer, we discover new hybrid LLM architectures that outperform Llama 3.2. Compared to Llama 3.2 and previous state-of-the-art baselines, the new model architectures consistently reduce validation loss at parameter scales of 350M-3B and improve evaluation accuracy on the downstream tasks by up to 2.8-8.3% (1.1-3.1% on average) while improving both training and inference efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。