arXiv:2608.29973cs.AIcs.LG2026-08

开源工具链实现比特币数据流处理,支持价格预测与链上欺诈检测。

An Open-Source, Event-Driven Pipeline for Cryptocurrency Market Data: Ingestion, Forecasting, and On-Chain Fraud Detection

  • 用Kafka和文件监控替代云触发器,实现在普通硬件上的事件驱动数据流水线。
  • 基于历史数据对比了ARIMA与LSTM在比特币价格预测中的表现,欺诈检测准确率超90%。
  • 适合区块链开发者、量化研究者及对低代码数据系统感兴趣的人群。

加密货币市场产生高频、多源数据,若无商用级流式处理与仓储基础设施则难以使用。本文描述了一个完全开源的数据管道,复现了云原生事件驱动系统的功能——文件到达触发消息,消息触发计算——全部运行于普通硬件,采用Apache Kafka和文件系统轮询器替代托管云触发机制。该管道将历史Gemini交易所数据按小时和分钟粒度分片,通过两个独立分组的Kafka消费者异步摄入(一个用于审计日志,一个触发Spark ETL),并将清洗后的结果存入含历史与聚合模式及资产特定数据集市的PostgreSQL仓库。利用生成的比特币数据集市,比较了季节性ARIMA模型与单层LSTM网络在价格预测上的效果;另针对Farrugia等人提出的公开以太坊欺诈检测基准,应用随机森林与梯度提升分类器并加入特征工程。报告了系统架构、建模方法与结果指标,并明确指出在不同预测时序下比较预测性能的局限性,以及在静态已标注数据集上评估欺诈检测的约束。

原文摘要 · Abstract (English)

Cryptocurrency markets generate high-frequency, multi-source data that is expensive to work with unless a team already has commercial-grade streaming and warehousing infrastructure in place. This paper describes a fully open-source pipeline that reproduces the behavior of a cloud-native, event-driven system -- file arrival triggering a message, a message triggering compute -- entirely on commodity hardware, using Apache Kafka and a filesystem-watching poller in place of managed cloud triggers. The pipeline partitions historical Gemini exchange data into hourly and minutely files, ingests them asynchronously through two independently grouped Kafka consumers (one for audit logging, one for Spark-triggered ETL), and lands cleaned output in a PostgreSQL warehouse with historical and aggregated schemas plus asset-specific data marts. We use the resulting Bitcoin data mart to compare a seasonal ARIMA model against a single-layer LSTM network for price forecasting, and separately apply Random Forest and Gradient Boosting classifiers, with additional engineered features, to the public Ethereum fraud detection benchmark introduced by Farrugia et al. We report the architecture, the modeling methodology, and the resulting metrics, and we are explicit about the limitations of comparing forecasts issued at different horizons and of evaluating fraud detection on a static, already-labeled dataset.

数据管道区块链欺诈检测时间序列

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。