用符号化张量图生成大规模分布式LLM训练模拟，支持高效系统优化。
Scalable Synthesis of distributed LLM workloads through Symbolic Tensor Graphs
- 基于符号化张量图构建分布式推理与训练工作负载
- 可模拟超12.8万GPU规模，保持计算/内存/通信精度
- 适合研究模型并行策略与软硬件协同设计的团队
在大规模AI训练与推理系统中，优化大语言模型（LLM）性能需要一种可扩展且表达力强的方法来建模分布式工作负载执行。此类建模对于部署前的系统级优化（如并行策略）和硬件设计空间探索至关重要。尽管近期已有研究尝试从真实系统收集执行轨迹，但大规模基础设施仅对主要云服务商开放，且特定平台的轨迹难以适应其他软硬件配置，尤其在大规模场景下。我们提出STAGE框架，通过合成高保真执行图，精确建模包括LLM和MoE在内的分布式AI工作负载。STAGE支持全面的并行化策略，使用户能够系统性地探索广泛的模型架构与系统配置组合。其可扩展性体现在能合成覆盖超过12.8万GPU的高保真LLM轨迹，同时在计算、内存和通信层面保持张量级精度。STAGE已开源：https://github.com/astra-sim/stage
原文摘要 · Abstract (English)
Optimizing the performance of large language models (LLMs) on large-scale AI training and inference systems requires a scalable and expressive mechanism to model distributed workload execution. Such modeling is essential for pre-deployment system-level optimizations (e.g., parallelization strategies) and hardware design-space explorations. While recent efforts have proposed collecting execution traces from real systems, access to large-scale infrastructure remains limited to major cloud providers. Moreover, traces capturing execution on a specific platform cannot be easily adapted to study alternate software and/or hardware configurations, especially at scale. We introduce STAGE, a framework that synthesizes high-fidelity execution graphs to accurately model distributed AI workloads (including LLMs and MoEs). STAGE supports a comprehensive set of parallelization strategies, allowing users to systematically explore a wide spectrum of model architectures and system configurations. STAGE demonstrates its scalability by synthesizing high-fidelity LLM traces spanning over 128K GPUs, while preserving tensorlevel accuracy in compute, memory, and communication. STAGE is publicy available at https://github.com/astra-sim/stage
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。