对比8种优化器在不同预算下的表现,找出适配场景的最优选择。
When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

- 系统性测试8种优化器在22个可配置系统上的表现。
- 模型驱动型优化器适合预算紧张,无模型方法在预算充足时更优。
- 新发现的FLASH优化器在各类系统中均表现稳定,适合通用场景。
软件配置调优对系统性能优化至关重要,过去十年出现了多种优化器。然而,不同系统在调优过程中的耗时差异显著:例如,PostgreSQL需数分钟测量一次配置,而MariaDB则可能需要数小时。此外,同一系统内用户预算和偏好也各不相同。这引出关键问题——给定预算水平,哪种优化器最适合系统工程实践?此问题重要,因为优化器常有其“舒适区”,在不同预算下表现迥异。本文系统评估了8种成熟优化器在22个可配置系统上于不同预算下的表现。结果表明,模型基优化器(如SMAC)在紧预算下表现优异,而模型无关优化器(如遗传算法)在预算宽松时更优。有趣的是,一种名为FLASH的优化器在多数系统中均表现稳定,不受预算影响。进一步分析发现,许多系统具有良好的局部最优解(吸引域大),使贪心类优化器(如FLASH)能取得出色效果。代码、数据及补充材料见 https://anonymous.4open.science/r/Config-W2W-98B2。
原文摘要 · Abstract (English)
Software configuration tuning is crucial for optimising system performance, and various optimisers have emerged over the last decade. Yet, the time required during the tuning process may vary across systems. In some systems (e.g., PostgreSQL), it may take a few minutes to measure a configuration, whereas in some others (e.g., MariaDB), it can take several hours. Moreover, even within the same system, users may have varying budgets and preferred settings. This naturally raises a question -- Given a budget level, which optimiser is the best choice for SE practitioners? This matters because optimisers usually have their own ``comfort zone'' and may perform very differently under distinct budgets. In this paper, we aim to answer this question. We systematically evaluate eight well-established optimisers across 22 configurable systems under varying budget levels. We find that, unsurprisingly, model-based optimisers (e.g., SMAC) are well-suited under tight budgets, and model-free optimisers (e.g., GAs) become superior with more generous budgets. However, interestingly, there is one optimiser, FLASH, that performs consistently well on most systems regardless of budgets. We lastly investigate the reasons behind this phenomenon and find that many systems possess good local optima (with large basins of attraction), allowing greedy optimisers (e.g., FLASH) to achieve strong performance. Source code, data, and supplementary materials of this work are available at https://anonymous.4open.science/r/Config-W2W-98B2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。