用频谱熵决定每令牌计算方式，节省算力同时保持性能。
Chiaroscuro Attention: Spending Compute in the Dark
- 根据令牌频谱复杂度动态选择快速或完整注意力计算。
- 在WikiText-103上减少35%-40%计算量，损失仅3.93点PPL。
- 适合需要高效推理的中等规模语言模型应用。
我们提出CHIAR-Former（基于明暗对比注意力的Transformer），通过每令牌频谱熵H(x)∈[0,1]判断其频率域复杂度，动态路由至DCT频谱混合（O(d log d)，次二次）或全自注意力（O(n²d)，序列长度平方）。主要贡献：(1) 发现路由坍缩现象——三操作器系统退化为DCT+注意力，揭示最优操作子集；(2) 提出任务级元路由g=σ(Linear(x̄))∈[0,1]，其中x̄为批次均值嵌入，实现频谱与身份路径的端到端软融合；(3) 在4亿参数下实现35%-40% FLOP削减，测试PPL为27.51（原模型23.58），代价仅3.93。在混合数据训练中，对小语料库显著优于全注意力，证实频谱混合具正则化价值。元路由稳定在g≈0.22，表明在大规模下模型达成稳健的算力-质量平衡：注意力层吸收表征复杂度，频谱预处理高效锚定低频结构。
原文摘要 · Abstract (English)
We introduce CHIAR-Former (CHIAroscuro Attention-based tRansFormer), an efficient transformer that routes each token to either DCT spectral mixing (O(d log d), sub-quadratic) or full self-attention (O(n^2 d), quadratic in sequence length n) based on per-token spectral entropy H(x) in [0,1], which measures the frequency-domain complexity of each token embedding x. We make three contributions: (1) we discover routing collapse -- a three-operator system collapses to DCT+Attention, revealing the optimal operator subset; (2) we propose a learned task-level MetaRouter g = sigma(Linear(x-bar)) in [0,1], where x-bar is the batch-mean embedding and g soft-blends spectral and identity paths end-to-end; and (3) we demonstrate 35-40% FLOP reduction at 400M parameters with a 3.93 PPL cost on WikiText-103 (Test PPL 27.51 vs. 23.58). Under mixed-dataset training, CHIAR-Former dramatically outperforms full attention on small corpora, confirming the regularisation value of spectral mixing. The MetaRouter stabilises at g ~ 0.22, indicating that at scale the model reaches a robust compute-quality equilibrium: attention layers absorb representational complexity while spectral preprocessing efficiently anchors low-frequency structure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。