用景观分析法发现:模型好用不只看准确率,还能自动选最优调参组合。
Evaluating Useful Surrogate Models for Configuration Tuning Beyond Accuracy: A Fitness Landscape Analysis Perspective
- 以适应度景观视角替代准确率评估模型实用性
- 在27,000个案例中验证,90%以上场景下优于随机选择
- 提出Model4Tune工具,无需实测即可推荐最佳模型-调参组合
为高效优化部署与维护阶段的软件系统性能(如延迟),许多调参工具采用代理模型加速过程,而非直接依赖代价高昂的系统测量。尽管通常认为模型越准确越好,但先前研究发现‘准确率可能误导’,引发对代理模型实际作用的疑问。本文首次系统探索并讨论这一问题,基于适应度景观分析视角提出替代准确率的模型实用性评估理论,并通过涵盖最多27,000个案例的实证研究加以验证。据此,我们提出Model4Tune——一个自动化预测工具,可在不进行昂贵调参探测的前提下,估计未知系统中最优的模型-调参组合。实验表明,Model4Tune在79%–82%的案例中显著优于随机猜测,极大降低了软件系统配置工程的工作量。研究成果不仅揭示未来研究方向,也为实践者提供可落地的评估方案。
原文摘要 · Abstract (English)
To efficiently tune configuration for better software system performance (e.g., latency) at the deployment and maintenance stage, many tuners have leveraged a surrogate model to expedite the process instead of solely relying on the profoundly expensive system measurement. As such, it is naturally believed that we need more accurate models. However, the fact of "accuracy can lie"-a somewhat surprising finding from prior work-has left us many unanswered questions regarding what role the surrogate model plays in configuration tuning. This paper provides the very first systematic exploration and discussion, together with a resolution proposal, to disclose the many faces of useful surrogate models for configuration tuning beyond accuracy, through the novel perspective of fitness landscape analysis. We present a theory as an alternative to accuracy for assessing the model usefulness in tuning, based on which we conduct an extensive empirical study involving up to 27,000 cases. Drawing on the above, we propose Model4Tune, an automated predictive tool that estimates which model-tuner pairs are the best for an unforeseen system without expensive tuner profiling. Our results suggest that Model4Tune, as one of the first of its kind, performs significantly better than random guessing in 79%-82% of the cases, hence greatly mitigating the required efforts in engineering configuration for software systems. Our results not only shed light on the possible future research directions but also offer a practical resolution that can assist practitioners in evaluating the most useful model for configuration tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。