用相似度引导的渐进式微调,提升大模型生成新神经网络架构的能力。
Similarity-Guided Curriculum Fine-Tuning of LLMs for Neural Architecture Synthesis
- 基于MinHash构建代码相似度分层,逐步增加架构多样性进行训练。
- 在CIFAR-10上达到60%最高成功率,无需后期修复。
- 揭示接口修复与课程调度针对不同失败模式,适合架构生成研究者。
提出一种基于MinHash的相似度调度框架,对基于大模型的神经架构搜索(NAS)中的架构代码构建渐进式课程。通过128次置换的MinHash签名对归一化的7-gram源码片段进行处理,将参考池划分为相似度带,并按架构异质性递增顺序呈现;每个阶段的最佳LoRA适配器累积合并至主干模型。在OlympicCoder-7B上,于LEMUR基准下对CIFAR-10图像分类任务进行评估,每轮生成N=15个候选架构,共经历六步渐进微调。该课程在高相似度层级实现60%峰值成功率,无需后处理修复。在最多样化的层级进行2×2消融实验:无修复时,基础模型(47%峰值SR)显著优于课程模型(7% SR);加入部分接口修复后,两者均达53%峰值SR。该现象表明,合并级权重漂移逐渐抹除评估器-接口先验,说明接口修复与课程调度应对不同失效模式。此外,在SVHN数据集上观察到跨数据集迁移:不使用课程预热的基础模型生成,仅达27%峰值成功率,且准确率显著更低(60.5%),与unq家族锚点架构合成难度增加一致。
原文摘要 · Abstract (English)
Introduce a MinHash-based similarity scheduling framework that constructs a progressive curriculum over neural architecture code for LLM-based neural architecture search (NAS). Using 128-permutation MinHash signatures over normalised 7-gram source code shingles, we partition the reference pool into similarity bands and present them in increasing architectural heterogeneity, with the best LoRA adapter from each stage merged cumulatively into the backbone. We evaluate the framework on OlympicCoder-7B within the LEMUR benchmark on CIFAR-10 image classification, generating N =15 candidate architectures per epoch across six progressive fine-tuning steps. The curriculum achieves 60% peak success rate at the high-similarity level without post-processing repair. A 2*2 ablation at the most diverse level curriculum versus base model, with versus without partial interface repair reveals that without repair the base model (47% peak SR) substantially outperforms the curriculum model (7% SR), while adding partial repair brings both to 53% SR. This pattern is consistent with merge-level weight drift progressively erasing evaluator-interface priors, and suggests that interface repair and curriculum scheduling target distinct failure modes. We further report a cross-dataset transfer observation on SVHN, where direct base-model generation without curriculum warmup yields 27% peak SR at substantially lower accuracy (60.5%) than the CIFAR-10 equivalent, consistent with the increased synthesis difficulty of the unq-family anchor architecture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。