提出零数据搜索方法，高效找到适合Transformer的优秀模型结构。
ZeroLM: Data-Free Transformer Architecture Search for Language Models
- 通过分解Transformer模块并统计权重，快速评估模型能力。
- 在FlexiBERT上相关性达Spearman's rho 0.76，优于传统方法。
- 计算极快，适合大规模语言模型架构搜索，无需训练数据。
神经架构搜索（NAS）虽能自动化设计神经网络结构，但因计算成本过高难以普及。现有零成本代理方法在Transformer模型上的排序性能不佳，常不如简单参数量度量。当前自动代理发现方法存在搜索时间长、易过拟合和结构复杂等问题。本文提出一种新型零成本代理方法，通过高效计算权重统计信息，并将Transformer架构分解为功能不同的子模块，优化各模块对整体性能的贡献平衡。全面评估显示，该方法在FlexiBERT基准上达到Spearman's rho 0.76 和 Kendall's tau 0.53。所提方法计算效率极高，在多种NAS基准任务中保持稳健性能，为大规模架构搜索提供了实用解决方案。
原文摘要 · Abstract (English)
Neural architecture search (NAS) provides a systematic framework for automating the design of neural network architectures, yet its widespread adoption is hindered by prohibitive computational requirements. Existing zero-cost proxy methods, while reducing search overhead, demonstrate inadequate performance in architecture ranking tasks, particularly for Transformer-based models where they often underperform simple parameter counting metrics. Current automated proxy discovery approaches suffer from extended search times, susceptibility to data overfitting, and structural complexity. This paper introduces a novel zero-cost proxy methodology that quantifies model capacity through efficient weight statistics computation while decomposing Transformer architectures into functionally distinct sub-modules, thereby optimizing the balance of their contributions to overall performance. Our comprehensive evaluation demonstrates the superiority of this approach, achieving a Spearman's rho of 0.76 and Kendall's tau of 0.53 on the FlexiBERT benchmark. The proposed method exhibits exceptional computational efficiency while maintaining robust performance across diverse NAS benchmark tasks, offering a practical solution for large-scale architecture search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。