arXiv:2608.05944cs.DCcs.LG2026-08被引 1

在NVIDIA B300上实测327亿参数模型全量微调,分享硬件级运维经验。

Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening

论文配图:Operating Multi-Node Full Fine-Tuning on NVIDIA B300: A Field Report on Telemetry-Based Triage, Negative Results, and Operational Hardening
图 1 · 摘自论文原文
  • 通过功耗差异识别计算/通信/数据饥饿等故障模式,实现精准诊断。
  • 实测发现本地缓存与NFS读取速度相当,且曾误判的吞吐下降实为资源竞争所致。
  • 提供可复用的故障检测门控和监控方案,避免数小时无声失败。

我们报告了在16块NVIDIA B300(两节点,使用FSDP / ZeRO-3)上对32.76B参数稠密模型Qwen3-32B进行全量微调的实操经验——这是首批公开的关于该加速器的现场报告。本文不提出新算法,仅整合现有实践并提供校准后的测量数据。具体贡献包括:(1) 一份基于板卡功耗的故障分类表,可区分计算、通信、数据饥饿、检查点或死锁、空闲状态(利用率达100%时可能为NCCL阻塞);(2) 一组坦诚的负面结果:控制实验显示,每步NFS读取速度约53k tok/s,与预分词本地缓存相当,因语料可装入页缓存且任务为计算密集型;另一次“吞吐崩溃”被还原为NFS与CPU争用问题而非存储介质瓶颈;(3) 在B300上4/8/16卡的强缩放性能与GPU小时数数据,近线性表现,提供绝对参考值;(4) 一个实际故障案例:因每秩令牌打包不平衡导致的训练轮末尾NCCL死锁,配套2.7秒预运行不变量检查与外部观察器,将多小时无声失败转为即时拒绝。该解法对应PyTorch的文档化规范‘Join / equalize-to-minimum’,我们报告其带来的代价与节省的GPU小时数。核心启示是:对数据依赖的数据并行任务,应关注功耗而非利用率,启动前验证不变量——通过冒烟测试不代表可安全运行。

原文摘要 · Abstract (English)

We report operational experience full-fine-tuning a 32.76B-parameter dense model (Qwen3-32B) on 16 x NVIDIA B300 (two nodes, FSDP / ZeRO-3) -- among the first published field accounts on this accelerator. We claim no new algorithm. The individual mechanisms we use are established practice; our contribution is the integrated field experience and a set of calibrated measurements on new hardware. Concretely we offer four practitioner artifacts. (1) A B300-calibrated power-draw triage table that distinguishes compute / communication / data-starvation / checkpoint-or-deadlock / idle by board wattage (utilization% reads 100% during an NCCL hang). (2) A set of honest negative results that dispel common optimization folklore at this scale: a controlled A/B in which per-step NFS reading matches a pretokenized local cache (~53k tok/s) because the corpus fits in page cache and the job is compute-bound; and a reconstruction of an earlier "throughput collapse" as NFS/CPU contention rather than a storage-medium limit. (3) Calibrated 4/8/16-GPU strong-scaling and GPU-hour numbers on B300 (near-linear, as expected in this regime; we report absolute values as reference data). (4) A worked failure case -- an epoch-end NCCL deadlock from per-rank token-packing imbalance -- together with a 2.7-second pre-run invariant gate and an external watcher that turn multi-hour silent failures into instant rejections. This deadlock and its remedy correspond to PyTorch's documented Join / equalize-to-minimum practice; we position our instantiation against that prior art and report the GPU-hours the failure cost and the gate saves. The transferable takeaway is operational, not algorithmic: for data-dependent data-parallel jobs, watch power rather than utilization, and verify invariants before launch -- a passing smoke test is not evidence of a safe full run.

大模型微调硬件运维故障诊断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。