用代码差分精调大模型，高效生成神经网络架构。
Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffs
- 大模型不从零生成完整代码，而是输出修改基线架构的紧凑差分包。
- 生成准确率提升至65.8%，代码量减少75%-85%。
- 适合需要快速迭代架构、资源有限的研究者使用。
大型语言模型（LLMs）在神经架构生成中展现出巨大潜力，但现有方法需从头生成完整模型实现，计算成本高且代码冗长。本文提出Delta-Code Generation，通过微调的LLM生成统一的紧凑代码差分（delta）来优化基线架构，而非合成整个模型。该流程利用LEMUR数据集中的精选架构，采用LoRA进行迭代微调，并结合MinHash-Jaccard新颖性过滤以保证结构多样性。评估涵盖三款7B级LLM（DeepSeek-Coder-7B、Qwen2.5-Coder-7B、Mistral-7B），在六大数据集（CIFAR-10、CIFAR-100、MNIST、SVHN、ImageNette、CelebA）上进行22轮测试（每模型1,100个候选）。所有模型显著优于全生成基线（有效率50.6%，首轮平均准确率42.3%）：DeepSeek-Coder达75.3%有效率与65.8%平均准确率；Qwen2.5-Coder为72.1%/64.6%；Mistral为66.6%/66.1%。在CIFAR-10上，最佳首轮准确率达85.5%（Mistral）、85.2%（DeepSeek）、80.6%（Qwen），远超全生成的63.98%及同期方法71.5%。代码长度由200+行降至30-50行，压缩率达75%-85%。50轮实验验证首轮代理性能可保持排名一致性（如Mistral Spearman ρ=0.926）。Delta-based生成是一种高效、跨领域、兼容多种大模型的替代方案。
原文摘要 · Abstract (English)
Large language models (LLMs) show strong potential for neural architecture generation, yet existing approaches produce complete model implementations from scratch -- computationally expensive and yielding verbose code. We propose Delta-Code Generation, where fine-tuned LLMs generate compact unified diffs (deltas) to refine baseline architectures rather than synthesizing entire models. Our pipeline iteratively fine-tunes the LLM via LoRA on curated architectures from the LEMUR dataset, with MinHash-Jaccard novelty filtering for structural diversity. We evaluate three 7B-class LLMs -- DeepSeek-Coder-7B, Qwen2.5-Coder-7B, and Mistral-7B -- across six datasets (CIFAR-10, CIFAR-100, MNIST, SVHN, ImageNette, CelebA) using a 22-cycle protocol (1,100 candidates per LLM). All three substantially surpass the full-generation baseline (50.6% valid rate, 42.3% mean first-epoch accuracy): DeepSeek-Coder reaches 75.3% valid rate and 65.8% mean accuracy; Qwen2.5-Coder 72.1%/64.6%; Mistral 66.6%/66.1%. On CIFAR-10, best first-epoch accuracies reach 85.5% (Mistral), 85.2% (DeepSeek), 80.6% (Qwen) -- well above 63.98% full generation and 71.5% for the concurrent approach of Gu et al. Output lengths are 30-50 lines versus 200+ for full generation (75-85% reduction). A 50-epoch study confirms the 1-epoch proxy preserves rankings (Mistral: Spearman $ρ$ = 0.926). Delta-based generation is a token-efficient, multi-domain, LLM-agnostic alternative to full-model synthesis for LLM-driven NAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。