用哈达玛变换替代注意力输出投影，显著降低参数和内存开销。
Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers
- 用固定哈达玛变换加对角仿射变换替代密集投影
- 每层减少约25%参数，训练时计算效率更高
- 大模型、长序列下优势更明显，适合高效部署
多头注意力中的密集输出投影随模型维度呈平方增长，显著增加参数量、内存占用和推理成本。我们提出用固定的、无参数的沃尔什-哈达玛变换（WHT）结合对角仿射变换替代该投影。该方法每层块约减少25%注意力参数，通过正交且保范变换维持跨头全局交互。实验表明，与密集基线相比，WHT增强模型在训练中验证损失下降更快，反映更高的计算利用率。关键的是，效率提升——包括更低内存占用和更高吞吐量——随模型规模、批大小和序列长度单调增长。我们在预填充和解码阶段均评估性能，发现随着复杂度上升，结构化变换始终优于密集投影。结果表明，在相同训练预算下，用结构化变换替代密集投影可实现更高效的架构并达到更低损失。
原文摘要 · Abstract (English)
The dense output projection in multi head attention scales quadratically with model dimension, contributing significantly to parameter count, memory footprint, and inference cost. We propose replacing this projection with a fixed, parameter free Walsh Hadamard Transform (WHT) followed by a diagonal affine transformation. This approach eliminates approximately 25 percent of attention parameters per block while maintaining global cross-head interaction through an orthogonal, norm-preserving transformation. Our results demonstrate that WHT augmented models exhibit a steeper validation loss curve relative to training FLOPs compared to dense baselines, suggesting superior compute utilization during training. Crucially, we show that efficiency gains including reduced memory footprint and increased throughput grow monotonically with model size, batch size, and sequence length. We evaluate performance across both prefill and decoding stages, finding that the structured transform consistently outperforms dense projections as complexity increases. Our findings indicate that replacing dense projections with structured transforms allows for more compute-efficient architectures that achieve lower loss than dense models at an equivalent training budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。