arXiv:2607.29383cs.LGcs.DC2026-07

用混合模型实时检测HDFS日志中的异常数据块,提升系统维护效率。

Exploring Block Anomaly Detection In HDFS Log Data Analysis

  • 构建LLM-BiLSTM混合模型分析日志模式
  • 通过Kafka实现日志流实时处理,支持秒级异常检测
  • 适合运维人员快速定位分布式存储故障

随着大数据技术发展,越来越多企业采用HDFS进行数据存储与处理,其维护成为数据管理的关键。随着服务器功能多样化、服务复杂化,日志记录了大量实时事件,帮助运维人员定位故障。然而,HDFS生成海量非结构化日志,逐条排查耗时且低效。本文提出一种流式HDFS日志块异常检测工作流:利用并行计算处理历史日志,构建基于LLM-BiLSTM的混合深度学习模型识别异常数据块,并基于Kafka搭建流式日志管道,实现对HDFS日志块异常的实时检测,显著提升故障定位速度与准确性。

原文摘要 · Abstract (English)

In recent years, with the development of big data technology, increasingly more companies use HDFS for data processing and storage. As a result, the maintenance of distributed file systems has become an extremely important part of data management. As the function of server systems is becoming increasingly diversified and their services are becoming complex, the logs, recording real-time events make it easier for system operators to locate the failures and errors that happened in the server systems to make server always available. HDFS, a distributed file system, which contains large data sets, will record a large number of logs. Moreover, the logs are not always structured data, they are not stable as well. However, to detect the problems that occur in the system by checking one log by one log, it's complicated and boring work for the system operators. Using machine learning techniques and natural language processing techniques to detect the HDFS block anomaly will help the system operators to locate and fix the anomaly rapidly and accurately. This paper proposes a streaming HDFS log block anomaly workflow. It helps maintenance practitioners to use parallel computing network in processing historical log, and construct LLM-BiLSTM hybrid deep learning model to detect anomaly block in HDFS, then build streaming log pipeline based on Kafka to give one real-time HDFS log block anomaly detection solution.

日志分析异常检测流处理HDFS

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。