针对FP4注意力计算瓶颈，提出直接映射与路径优化，提升推理和训练速度。
Hardware-Aware FP4 FlashAttention-4
- 直接映射分数到FP4概率，跳过软化转换
- 单卡80亿参数更新加速1.14倍，前向吞吐达BF16的2.13倍
- 适用于追求高吞吐的大型模型推理与训练场景
Blackwell的4位浮点（FP4）张量核心并未自动提升注意力计算速度，因为软化转换和片上依赖在矩阵乘法缩小后成为瓶颈。本文提出 extit{Direct-P}用于非因果推理，以及一种将前向量化直接传递至反向的因果路径。Direct-P将注意力分数直接映射为FP4概率，在NVIDIA GB200上实现高达2.13倍于bfloat16（BF16）的前向吞吐。因果路径通过保存的量化查询与键重构概率，并使用8位浮点（FP8）梯度操作，使单卡80亿参数的完整更新加速达1.14倍。分布式训练中保持FP8概率与值；所有测试的MXFP4概率/值训练轨迹均出现发散。
原文摘要 · Abstract (English)
Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with \emph{Direct-P} for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13$\times$ the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14$\times$. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。