arXiv:2605.16347cs.LG2026-05

专为高性能计算设计的智能助手,让科研人员轻松管理集群任务。

HPC-LLM: Practical Domain Adaptation and Retrieval-Augmented Generation for HPC Support

  • 用检索增强+轻量微调,让大模型懂HPC操作
  • 80亿参数模型性能逼近140亿参数大模型
  • 适合需要高效管理计算集群的科研人员

现代科学研究高度依赖高性能计算(HPC)基础设施,但研究人员在使用集群环境、作业调度器、GPU资源和并行计算框架时仍面临诸多操作障碍。通用大语言模型虽能提供编程帮助,却缺乏可靠的领域知识。本文提出HPC-LLM,一个针对常见HPC工作流(如Slurm调度、MPI执行、GPU利用、文件系统管理与集群故障排查)的检索增强型领域自适应助手。该框架整合了自动化文档摄入、密集检索、基于QLoRA的轻量级领域微调及本地推理的模块化流水线。我们构建了一个面向HPC的语料库,涵盖公开的高校HPC文档、经筛选的操作案例以及从检索内容生成的合成指令-答案对,共约9,000至24,000个训练样本,覆盖作业调度、GPU计算、分布式训练、存储系统与集群管理等主题。采用QLoRA对Llama 3.1 8B进行微调,并在JetStream2基础设施上评估其在检索增强设置下的表现。实验表明,经过微调的80亿参数模型在性能上可媲美更大规模的通用模型,同时显著降低显存占用与推理延迟。具体而言,其性能接近Qwen 2.5 14B,但所需计算资源大幅减少。

原文摘要 · Abstract (English)

Modern scientific research increasingly depends on High-Performance Computing (HPC) infrastructures, yet many researchers face significant operational barriers when interacting with cluster environments, job schedulers, GPU resources, and parallel computing frameworks. General-purpose large language models (LLMs) provide useful coding assistance but often lack the domain-specific operational knowledge required for reliable HPC support. This paper presents HPC-LLM, a retrieval augmented and domain-adapted assistant designed to support common HPC workflows including Slurm scheduling, MPI execution, GPU utilization, filesystem management, and cluster troubleshooting. The proposed framework integrates automated documentation ingestion, dense retrieval, lightweight domain adaptation using QLoRA, and local inference within a modular orchestration pipeline. To support domain adaptation, we construct an HPC-oriented corpus from publicly available university HPC documentation, curated operational examples, and synthetic instruction-answer pairs generated from retrieved HPC content. The resulting dataset contains approximately 9,000 to 24,000 HPC-focused training examples spanning job scheduling, GPU computing, distributed training, storage systems, and cluster administration topics. We fine-tune Llama 3.1 8B using QLoRA and evaluate the resulting model against several open weight baselines under retrieval-augmented settings on JetStream2 infrastructure. Experimental results indicate that the adapted 8B model achieves performance comparable to substantially larger general-purpose models while operating under significantly lower GPU memory requirements and inference latency. In particular, the adapted model approaches the performance of Qwen 2.5 14B while requiring substantially fewer computational resources.

HPC大模型微调检索增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。