FeatInsight解决线上特征管理瓶颈,实现万亿维特征毫秒级更新。
FeatInsight: An Online ML Feature Management System on 4Paradigm Sage-Studio Platform
- 基于OpenMLDB引擎构建全生命周期特征管理系统
- 支持万亿维特征空间,特征更新延迟低至毫秒级
- 适合需要实时特征迭代的推荐与风控场景
特征管理对在线机器学习应用至关重要,常成为性能瓶颈(如销售预测服务中占总延迟达70%)。不当特征配置会严重削弱模型泛化能力。由于原始数据规模大(如2018 PHM数据集含17张表、数十至数百列)、特征间依赖复杂且需高性能一致计算、同时要求快速更新以应对实时数据变化,特征管理极具挑战。本文演示了FeatInsight系统,支持特征设计、存储、可视化、计算、验证及血缘管理全生命周期。该系统基于OpenMLDB执行引擎,已在4Paradigm Sage Studio平台部署超过100个真实场景,处理高达万亿维特征空间,支持毫秒级特征更新。演示展示了其在在线商品推荐中提升设计效率、在在线欺诈检测中优化计算性能的效果。代码已开源:https://github.com/4paradigm/FeatInsight。
原文摘要 · Abstract (English)
Feature management is essential for many online machine learning applications and can often become the performance bottleneck (e.g., taking up to 70% of the overall latency in sales prediction service). Improper feature configurations (e.g., introducing too many irrelevant features) can severely undermine the model's generalization capabilities. However, managing online ML features is challenging due to (1) large-scale, complex raw data (e.g., the 2018 PHM dataset contains 17 tables and dozens to hundreds of columns), (2) the need for high-performance, consistent computation of interdependent features with complex patterns, and (3) the requirement for rapid updates and deployments to accommodate real-time data changes. In this demo, we present FeatInsight, a system that supports the entire feature lifecycle, including feature design, storage, visualization, computation, verification, and lineage management. FeatInsight (with OpenMLDB as the execution engine) has been deployed in over 100 real-world scenarios on 4Paradigm's Sage Studio platform, handling up to a trillion-dimensional feature space and enabling millisecond-level feature updates. We demonstrate how FeatInsight enhances feature design efficiency (e.g., for online product recommendation) and improve feature computation performance (e.g., for online fraud detection). The code is available at https://github.com/4paradigm/FeatInsight.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。