用大模型自动设计图像描述模型,无需人工干预。
LLM as a Neural Architect: Controlled Generation of Image Captioning Models Under Strict API Contracts
- 大模型根据严格API规范组合编码器与解码器生成新模型。
- 在MS COCO上生成数十个模型,超半数成功训练并产出有意义描述。
- 首次实现提示词驱动的架构生成与自动评估闭环,适合自动化机器学习研究者。
传统神经网络架构搜索(NAS)依赖大量人工经验或自动化试错。本文提出NN-Caption,一个由大模型引导的架构搜索流程,通过组合LEMUR分类模型的CNN编码器与序列解码器(LSTM/GRU/Transformer),在严格Net API约束下生成可运行的图像描述模型。以DeepSeek-R1-0528-Qwen3-8B为主生成器,给出提示模板与生成样例。在MS COCO数据集上评估,使用BLEU-4指标。大模型生成数十个描述模型,其中超过一半成功训练并生成有意义的描述。分析了提示中输入模型片段数量(5个与10个)的影响,发现提供更多候选组件时成功率略有下降。报告了训练动态(准确率随训练轮次变化)及最高达到的BLEU-4分数。结果表明,大模型不仅能提出架构,还能建议超参数与训练策略。识别出代码幻觉和API合规性等挑战,并通过提示规则与迭代修复解决。本工作构建了一个提示驱动的代码生成与自动评估集成管道,向开放的LEMUR数据集新增数十个新型描述模型,支持可复现的基准测试与下游AutoML研究。
原文摘要 · Abstract (English)
Neural architecture search (NAS) traditionally requires significant human expertise or automated trial-and-error to design deep learning models. We present NN-Caption, an LLM-guided neural architecture search pipeline that generates runnable image-captioning models by composing CNN encoders from LEMUR's classification backbones with sequence decoders (LSTM/GRU/Transformer) under a strict Net API. Using DeepSeek-R1-0528-Qwen3-8B as the primary generator, we present the prompt template and examples of generated architectures. We evaluate on MS COCO with BLEU-4. The LLM generated dozens of captioning models, with over half successfully trained and producing meaningful captions. We analyse the outcomes of using different numbers of input model snippets (5 vs. 10) in the prompt, finding a slight drop in success rate when providing more candidate components. We also report training dynamics (caption accuracy vs. epochs) and the highest BLEU-4 attained. Our results highlight the promise of LLM-guided NAS: the LLM not only proposes architectures but also suggests hyperparameters and training practices. We identify the challenges encountered (e.g., code hallucinations or API compliance issues) and detail how prompt rules and iterative code fixes addressed them. This work presents a pipeline that integrates prompt-based code generation with automatic evaluation, and adds dozens of novel captioning models to the open LEMUR dataset to facilitate reproducible benchmarking and downstream AutoML research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。