AutoScout自动优化ML系统配置,提升训练速度2.7至3倍。
AutoScout: Structured Optimization for Automating ML System Configuration
- 将配置问题建模为带层级依赖的混合离散连续优化问题。
- 在多种硬件和模型上实现2.7-3.0倍训练加速,优于人工调优。
- 适合需要自动化调优的机器学习系统开发者与工程师。
机器学习系统配置空间迅速扩大,涵盖模型并行策略、通信优化及底层运行时参数。系统整体性能高度依赖这些选择,但因特征类型异构(如稀疏与密集参数)、条件依赖关系(如特定上游决策下才有效的执行参数)以及高昂的搜索(性能测试)成本,高效配置难以确定。现有方法或仅优化有限维度,或依赖无法泛化的启发式规则。本文提出AutoScout,一种通用的ML训练、微调与推理系统配置工具。它将系统配置建模为具有层级依赖的混合离散/连续优化问题,并引入混合优化框架,协同优化稀疏结构决策与密集执行参数。为降低性能测试成本,AutoScout自适应优先处理高影响配置特征,并集成不同保真度的模拟器。在多种模型、硬件平台与部署目标下,AutoScout持续找到高性能配置,相比专家调优设置实现2.7–3.0×训练加速。
原文摘要 · Abstract (English)
Machine learning (ML) systems expose a rapidly expanding configuration space spanning model-parallelism strategies, communication optimizations, and low-level runtime parameters. End-to-end system efficiency is highly sensitive to these choices, yet identifying high-performance configurations is challenging due to heterogeneous feature types (e.g., sparse and dense parameters), conditional dependencies (e.g., valid execution parameters only under specific upstream decisions), and the high search (profiling) cost. Existing approaches either optimize a narrow subset of configuration dimensions or rely on ad-hoc heuristics that fail to generalize as configuration spaces continue to grow. We present AutoScout, a general-purpose systems configurator for ML training, fine-tuning, and inference. It formulates the system configuration as a mixed-discrete/continuous optimization problem with hierarchical dependencies and introduces a hybrid optimization framework that jointly refines sparse structural decisions and dense execution parameters. To reduce profiling cost, AutoScout adaptively prioritizes high-impact configuration features and ensembles simulators with varying fidelity. Across diverse models, hardware platforms, and deployment objectives, AutoScout consistently identifies high-performance configurations, achieving 2.7-3.0$\times$ training speedup over expert-tuned settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。