Kamae让Spark预处理流程无缝转为Keras模型，避免线上线下的数据偏差。
Kamae: Bridging Spark and Keras for Seamless ML Preprocessing
- 将PySpark预处理流程映射为等效Keras层，实现端到端一致
- 在MovieLens和Expedia真实场景中验证了流程一致性
- 适合需要跨训练推理环境保持预处理一致的工程师
在生产级推荐系统中，特征预处理必须在训练与推理环境中忠实复现，这通常需要在线下和线上环境重复编写逻辑，增加工程负担并引发数据分布偏移风险。我们提出Kamae，一个开源Python库，通过将PySpark预处理流水线转换为等效的Keras模型，弥合这一差距。Kamae提供一系列可配置的Spark转换器与估算器，每个均对应一个对应的Keras层，支持在整个机器学习生命周期中实现一致的端到端预处理。该框架在真实场景中得到验证，包括MovieLens数据集和Expedia的Learning-to-Rank流水线。代码已开源：https://github.com/ExpediaGroup/kamae。
原文摘要 · Abstract (English)
In production recommender systems, feature preprocessing must be faithfully replicated across training and inference environments. This often requires duplicating logic between offline and online environments, increasing engineering effort and introducing risks of dataset shift. We present Kamae, an open-source Python library that bridges this gap by translating PySpark preprocessing pipelines into equivalent Keras models. Kamae provides a suite of configurable Spark transformers and estimators, each mapped to a corresponding Keras layer, enabling consistent, end-to-end preprocessing across the ML lifecycle. Framework's utility is illustrated on real-world use cases, including MovieLens dataset and Expedia's Learning-to-Rank pipelines. The code is available at https://github.com/ExpediaGroup/kamae.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。