推荐系统评估中默认设置会隐藏地影响调参效果，导致结果不可靠。
The Hidden Cost of Defaults in Recommender System Evaluation
- 发现RecBole框架存在未文档化的早停策略，干扰随机与贝叶斯调参
- 实验显示默认行为带来的性能波动，可与不同搜索策略差异相当
- 适合关注实验可复现性与调参透明性的研究者与开发者
超参数优化对提升推荐系统性能至关重要，但其实施常被视为中立或次要问题。本文转向审计广泛使用的推荐框架RecBole，揭示其内部默认设置（特别是未文档化的早停策略）会提前终止随机搜索与贝叶斯优化，从而限制搜索覆盖范围且用户难以察觉。通过六种模型与两个数据集的对比，我们量化了不同搜索策略下的性能方差与搜索路径不稳定性。结果表明，隐藏的框架逻辑引入的变异性，可达不同搜索策略间差异的量级。这凸显了将框架视为实验设计主动组件的重要性，并呼吁开发更透明、可复现的推荐系统工具链。本文提出具体建议，帮助研究人员和开发者缓解隐藏配置行为，提升调参流程的透明度。
原文摘要 · Abstract (English)
Hyperparameter optimization is critical for improving the performance of recommender systems, yet its implementation is often treated as a neutral or secondary concern. In this work, we shift focus from model benchmarking to auditing the behavior of RecBole, a widely used recommendation framework. We show that RecBole's internal defaults, particularly an undocumented early-stopping policy, can prematurely terminate Random Search and Bayesian Optimization. This limits search coverage in ways that are not visible to users. Using six models and two datasets, we compare search strategies and quantify both performance variance and search path instability. Our findings reveal that hidden framework logic can introduce variability comparable to the differences between search strategies. These results highlight the importance of treating frameworks as active components of experimental design and call for more transparent, reproducibility-aware tooling in recommender systems research. We provide actionable recommendations for researchers and developers to mitigate hidden configuration behaviors and improve the transparency of hyperparameter tuning workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。