arXiv:2604.05168cs.AI2026-04被引 1

用微调大模型解析超算日志,高效发现系统异常与运行规律。

Instruction-Tuned LLMs for Parsing and Mining Unstructured Logs on Leadership HPC Systems

  • 用链式思维微调80亿参数模型,结合领域日志模板提升解析精度。
  • 在6亿条真实超算日志上验证,效果媲美700亿参数大模型。
  • 适合超算运维、系统分析人员,可本地部署且节能高效。

顶级超算系统生成海量异构且主要无结构的日志数据。由于日志来自不同软硬件及运行层,格式不一致,导致结构提取和模式发现极为困难。因此,高效的日志解析与挖掘对将原始遥测数据转化为可操作洞察至关重要,有助于揭示运行模式、诊断异常,并实现可靠、高效、可扩展的系统分析。近年来的大语言模型(LLM)为自动化理解超算日志提供了新方向。为此,我们提出一种领域适配的指令跟随型LLM框架,利用链式思维(CoT)推理实现高保真度的日志解析与结构化。该方法结合领域特定的日志模板数据与指令微调样本,对80亿参数的LLaMA模型进行微调,专用于超算日志分析。我们设计了一种混合微调策略,使通用大模型适应领域日志数据,实现隐私保护、本地部署、快速且低能耗的日志挖掘。在LogHub数据集上的实验表明,该方法解析准确率与更大模型(如LLaMA 70B和Anthropic Claude)相当。进一步通过解析前沿超算四周期间超过6亿条生产日志,成功揭示了时间动态特征、节点级异常以及工作负载-错误日志间的关联模式。

原文摘要 · Abstract (English)

Leadership-class HPC systems generate massive volumes of heterogeneous, largely unstructured system logs. Because these logs originate from diverse software, hardware, and runtime layers, they exhibit inconsistent formats, making structure extraction and pattern discovery extremely challenging. Therefore, robust log parsing and mining is critical to transform this raw telemetry into actionable insights that reveal operational patterns, diagnose anomalies, and enable reliable, efficient, and scalable system analysis. Recent advances in large language models (LLMs) offer a promising new direction for automated log understanding in leadership-class HPC environments. To capitalize on this opportunity, we present a domain-adapted, instruction-following, LLM-driven framework that leverages chain-of-thought (CoT) reasoning to parse and structure HPC logs with high fidelity. Our approach combines domain-specific log-template data with instruction-tuned examples to fine-tune an 8B-parameter LLaMA model tailored for HPC log analysis. We develop a hybrid fine-tuning methodology that adapts a general-purpose LLM to domain-specific log data, enabling privacy-preserving, locally deployable, fast, and energy-efficient log-mining approach. We conduct experiments on a diverse set of log datasets from the LogHub repository. The evaluation confirms that our approach achieves parsing accuracy on par with significantly larger models, such as LLaMA 70B and Anthropic's Claude. We further validate the practical utility of our fine-tuned LLM model by parsing over 600 million production logs from the Frontier supercomputer over a four-week window, uncovering critical patterns in temporal dynamics, node-level anomalies, and workload-error log correlations.

日志解析大模型应用超算系统链式思维

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。