用AI预测大数据任务资源需求，省钱又保性能。
Artificial Intelligence for Cost-Aware Resource Prediction in Big Data Pipelines
- 基于随机森林分析集群日志，提取资源使用特征进行预测。
- 预测准确率高达R²=0.99，平均误差仅0.0048，对中小任务效果极佳。
- 适合需要降本增效的云平台运维人员和自动化系统设计者。
现代云计算中高效资源分配是关键挑战。过度配置导致成本浪费，配置不足则可能引发性能下降和SLA违规。本文提出一种基于人工智能的资源利用率预测方法，采用随机森林回归模型分析谷歌Borg集群日志。通过对数据进行清洗、转换与特征提取（包括CPU、内存及使用分布），模型实现了高精度预测（R²=0.99，MAE=0.0048，RMSE=0.137），有效捕捉工作负载特征与资源消耗间的非线性关系。误差分析显示，该方法在中小型任务上表现优异，而大型罕见任务存在更高方差。结果表明，该AI驱动的预测方法可支持云环境中的成本感知自动伸缩，既减少冗余配置，又保障服务质量。
原文摘要 · Abstract (English)
Efficient resource allocation is a key challenge in modern cloud computing. Over-provisioning leads to unnecessary costs, while under-provisioning risks performance degradation and SLA violations. This work presents an artificial intelligence approach to predict resource utilization in big data pipelines using Random Forest regression. We preprocess the Google Borg cluster traces to clean, transform, and extract relevant features (CPU, memory, usage distributions). The model achieves high predictive accuracy (R Square = 0.99, MAE = 0.0048, RMSE = 0.137), capturing non-linear relationships between workload characteristics and resource utilization. Error analysis reveals impressive performance on small-to-medium jobs, with higher variance in rare large-scale jobs. These results demonstrate the potential of AI-driven prediction for cost-aware autoscaling in cloud environments, reducing unnecessary provisioning while safeguarding service quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。