XAutoLM用元学习加速语言模型微调，省时70%以上。
XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML
- 基于过往经验提取元特征，智能筛选高效微调配置。
- 在6个任务中平均提速4.5倍，搜索误差降低7倍。
- 适合追求高效、绿色的NLP模型微调研究者使用。
机器学习专家常依赖领域知识进行模型选择、超参数优化和资源分配。这对语言模型微调尤为重要，因反复实验带来巨大计算开销与环境影响。现有自动化框架无法同时解决整个模型选择与超参数优化问题。本文提出XAutoLM，一种融合元学习与AutoML的框架，通过复用历史成功与失败经验，高效优化判别与生成式语言模型的微调流程。该框架提取任务与系统级元特征，引导采样避开低效配置。在四个文本分类与两个问答基准上，XAutoLM在五个任务中超越零样本优化器的最高F1值，平均评估时间减少最多4.5倍，搜索误差比降低最多七倍，并发现比零样本帕累托前沿多达50%的优质微调方案。相较之下，简单记忆基线出现负迁移。我们已开源XAutoLM及经验存储库，推动自然语言处理领域的资源高效、绿色微调发展。
原文摘要 · Abstract (English)
Experts in machine learning leverage domain knowledge to navigate decisions in model selection, hyperparameter optimization, and resource allocation. This is particularly critical for fine-tuning language models (LMs), where repeated trials incur substantial computational overhead and environmental impact. However, no existing automated framework simultaneously tackles the entire model selection and hyperparameter optimization (HPO) task for resource-efficient LM fine-tuning. We introduce XAutoLM, a meta-learning-augmented AutoML framework that reuses past experiences to optimize discriminative and generative LM fine-tuning pipelines efficiently. XAutoLM learns from stored successes and failures by extracting task- and system-level meta-features to bias its sampling toward valuable configurations and away from costly dead ends. On four text classification and two question-answering benchmarks, XAutoLM surpasses zero-shot optimizer's peak F1 on five of six tasks, cuts mean evaluation time of pipelines by up to 4.5x, reduces search error ratios by up to sevenfold, and uncovers up to 50% more pipelines above the zero-shot Pareto front. In contrast, simpler memory-based baselines suffer negative transfer. We release XAutoLM and our experience store to catalyze resource-efficient, Green AI fine-tuning in the NLP community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。