用多保真度实验+人工干预，高效优化多任务语言模型超参。
Interim Report on Human-Guided Adaptive Hyperparameter Optimization with Multi-Fidelity Sprints
- 分阶段进行贝叶斯优化，先低精度快速筛除无效参数
- 逐步提升模型精度，结合超带剪枝提升效率
- 适合需要快速迭代的NLP模型调优研究者
本案例研究采用分阶段超参数优化流程，对比使用多阶段学习率调度和优化器参数分组的多任务自然语言模型变体。通过短时贝叶斯优化会话，结合多保真度、超参数空间剪枝、渐进式减半和一定程度的人工指导。使用Optuna TPE采样器与Hyperband剪枝器，以及Scikit-Learn高斯过程最小化。初期采用高效的低保真度试验剪枝超参数空间，后续阶段逐步提升模型保真度并应用Hyperband剪枝以提高效率。另一重点是利用元学习器调整推理阶段分类概率的阈值。方法在Eberts和Ulges于2021年提出的联合实体与关系抽取模型的一系列变体上进行了验证。
原文摘要 · Abstract (English)
This case study applies a phased hyperparameter optimization process to compare multitask natural language model variants that utilize multiphase learning rate scheduling and optimizer parameter grouping. We employ short, Bayesian optimization sessions that leverage multi-fidelity, hyperparameter space pruning, progressive halving, and a degree of human guidance. We utilize the Optuna TPE sampler and Hyperband pruner, as well as the Scikit-Learn Gaussian process minimization. Initially, we use efficient low-fidelity sprints to prune the hyperparameter space. Subsequent sprints progressively increase their model fidelity and employ hyperband pruning for efficiency. A second aspect of our approach is using a meta-learner to tune threshold values to resolve classification probabilities during inference. We demonstrate our method on a collection of variants of the 2021 Joint Entity and Relation Extraction model proposed by Eberts and Ulges.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。