arXiv:2602.23556cs.LGcs.AI2026-02中稿 · the 40th ACM Inter…

用大模型动态调度图神经网络预取,大幅减少通信开销。

Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents

  • 用大语言模型实现自适应预取,实时响应图数据变化。
  • 在超算上实测性能提升91%,通信量减少超50%。
  • 适合分布式图学习系统开发者与高性能计算研究者。

大规模图神经网络通常通过固定距离采样顶点邻居进行训练。由于输入图数据分布于多机,训练需频繁进行不规则通信,导致进程停滞。此外,获取的数据随图结构、分布方式、采样参数和缓存策略动态变化,静态预取方法难以适应。本文提出Rudder,一个嵌入当前主流AWS DistDGL框架的软件模块,可自主预取远程节点以最小化通信开销。其自适应机制不同于传统启发式或机器学习分类器。我们发现当代大语言模型(LLMs)具备涌现特性,如上下文学习(In-Context Learning, ICL)和多步逻辑推理能力,即使在严重欠训练情况下也表现良好,适用于动态控制任务。在标准数据集及未见过的配置下,于NERSC Perlmutter超算上的评估显示,相比无预取的基线DistDGL,Rudder实现高达91%的端到端训练性能提升;相较静态预取,性能提升82%,通信量减少超过50%。代码已开源:https://github.com/aishwaryyasarkar/rudder-llm-agent。

原文摘要 · Abstract (English)

Large-scale Graph Neural Networks (GNNs) are typically trained by sampling a vertex's neighbors to a fixed distance. Because large input graphs are distributed, training requires frequent irregular communication that stalls forward progress. Moreover, fetched data changes with graph, graph distribution, sample and batch parameters, and caching polices. Consequently, any static prefetching method will miss crucial opportunities to adapt to different dynamic conditions. In this paper, we introduce Rudder, a software module embedded in the state-of-the-art AWS DistDGL framework, to autonomously prefetch remote nodes and minimize communication. Rudder's adaptation contrasts with both standard heuristics and traditional ML classifiers. We observe that the generative AI found in contemporary Large Language Models (LLMs) exhibits emergent properties like In-Context Learning (ICL) for zero-shot tasks, with logical multi-step reasoning. We find this behavior well-suited for adaptive control even with substantial undertraining. Evaluations using standard datasets and unseen configurations on the NERSC Perlmutter supercomputer show up to 91% improvement in end-to-end training performance over baseline DistDGL (no prefetching), and an 82% improvement over static prefetching, reducing communication by over 50%. Our code is available at https://github.com/aishwaryyasarkar/rudder-llm-agent.

图神经网络分布式训练大模型应用预取优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。