用代码预测神经网络在不同数据集上的表现，验证了大模型可从代码中提取更强的判别信息。
From Code to Prediction: Fine-Tuning LLMs for Neural Network Performance Classification in NNGPT
- 通过微调大模型，仅凭网络代码预测其在不同数据集上的性能优劣
- 代码仅提示下模型达到80%准确率，显著高于依赖元数据的70%
- 代码比数据集属性蕴含更丰富的性能判别信号，适合架构分析研究
自动化机器学习框架越来越多地利用大语言模型（LLM）进行超参数优化和神经网络架构代码生成。然而，现有基于LLM的方法主要关注生成性输出，并通过训练生成结果来评估性能。大模型能否跨数据集推理神经网络表现仍缺乏探索。本文在NNGPT框架中构建了一个分类任务：微调后的LLM预测给定神经网络架构在两个图像分类数据集中哪个表现更好。该任务基于LEMUR数据集，提供标准化的PyTorch实现和可复现的性能指标。评估三种难度递增的提示配置：归一化准确率基准（100%）、使用数据集属性替换准确率的元数据提示，以及仅提供架构源代码和数据集名称的代码仅提示。使用DeepSeek-Coder-7B-Instruct配合LoRA微调，代码仅提示在15个周期内达到80%峰值准确率，元数据提示峰值为70%。分数据集分析显示互补优势：元数据在具有明显特征的数据集（如CelebAGender达90.9%）上表现更佳，但在特征重叠时下降；而代码仅提示表现更均衡。与DeepSeek-Coder1.3B对比表明模型容量影响此类架构推理能力。结果表明，大模型可通过微调从神经网络代码中预测跨数据集适用性，说明代码包含比数据集元数据更丰富的判别信号。
原文摘要 · Abstract (English)
Automated Machine Learning (AutoML) frameworks increasingly leverage Large Language Models (LLMs) for tasks such as hyperparameter optimization and neural architecture code generation. However, current LLM-based approaches focus on generative outputs and evaluate them by training the produced artifacts. Whether LLMs can learn to reason about neural network performance across datasets remains underexplored. We present a classification task integrated into the NNGPT framework, in which a fine-tuned LLM predicts which of two image classification datasets a given neural network architecture achieves higher accuracy on. The task is built on the LEMUR dataset, which provides standardized PyTorch implementations with reproducible performance metrics. Three prompt configurations of increasing difficulty are evaluated: a normalized-accuracy baseline (trivially reaching 100%), a metadata-enriched prompt replacing accuracies with dataset properties, and a code-only prompt presenting only architecture source code and dataset names. Using DeepSeek-Coder-7B-Instruct fine-tuned with LoRA, the code-only prompt reaches 80% peak accuracy over 15 epochs, while the metadata prompt peaks at 70%. Perdataset analysis reveals complementary strengths: metadata excels for datasets with distinctive properties (CelebAGender at 90.9%) but degrades for overlapping characteristics, whereas the code-only prompt shows more balanced performance. A comparison with DeepSeek-Coder1.3B confirms that model capacity affects this form of architectural reasoning. The results establish that LLMs can be fine-tuned to predict cross-dataset suitability from neural network code, suggesting that architecture source code contains richer discriminative signal than dataset metadata alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。