用小模型加专家模块，高效完成超参调优
Small LLMs with Expert Blocks Are Good Enough for Hyperparamter Tuning
- 设计轨迹摘要模块，让小模型理解训练过程
- 10次试验内性能接近GPT-4，误差小于0.9%
- 适合资源有限但需高效调参的研究者
超参数调优（HPT）是机器学习流程中的必要步骤，但随着模型规模增大，其计算成本上升且过程不透明。尽管已有研究探索使用大语言模型进行HPT，但多数依赖参数量超过1000亿的模型。本文提出一种基于小语言模型的专家块框架，核心为确定性轨迹上下文摘要器（TCS），可将原始训练轨迹转化为结构化上下文，使小模型具备与大模型相当的优化进程分析能力。实验中采用两个本地运行的小模型（phi4:reasoning14B 和 qwen2.5-coder:32B），在10次试验预算下，该框架在六个不同任务上的平均表现与GPT-4相差仅约0.9个百分点。
原文摘要 · Abstract (English)
Hyper-parameter Tuning (HPT) is a necessary step in machine learning (ML) pipelines but becomes computationally expensive and opaque with larger models. Recently, Large Language Models (LLMs) have been explored for HPT, yet most rely on models exceeding 100 billion parameters. We propose an Expert Block Framework for HPT using Small LLMs. At its core is the Trajectory Context Summarizer (TCS), a deterministic block that transforms raw training trajectories into structured context, enabling small LLMs to analyze optimization progress with reliability comparable to larger models. Using two locally-run LLMs (phi4:reasoning14B and qwen2.5-coder:32B) and a 10-trial budget, our TCS-enabled HPT pipeline achieves average performance within ~0.9 percentage points of GPT-4 across six diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。