FLARE++动态路由低秩注意力，提升不规则域微分方程代理模型性能。
FLARE++: Low-rank attention with dynamic attention routing
- 用输入条件生成动态查询，替代固定预设查询
- 在标准PDE代理任务上平均提升24%，长序列任务增2.3点准确率
- 支持多卡上下文并行，避免全序列聚集，适合大规模建模
全自注意力在不规则域的偏微分方程（PDE）代理中表现强劲，但其二次复杂度限制了高分辨率问题的应用。高效潜变量注意力模型如快速低秩注意力路由引擎（FLARE）通过将所有N个标记路由至M << N个学习的潜变量查询来规避该开销，但这些查询是固定参数：训练后对所有输入使用相同模板。FLARE++通过动态令牌路由打破这一限制，复用FLARE自身的编码器构建路由查询：学习的潜变量种子触发一次额外编码，将输入标记聚合为依赖输入的M个条件查询，并由这些查询决定如何压缩与重分配原始标记。这保持了FLARE的显式低秩分解和线性O(NM)复杂度，且仅用标准缩放点积注意力（SDPA）实现完整路由操作。我们还提供了多GPU上下文并行实现，将输入标记跨设备切分，从不在单个设备上汇聚完整序列。FLARE++在一系列标准PDE代理基准上表现优异，相较固定查询的FLARE平均提升24%，在长距离竞技场（Long Range Arena）上平均准确率提高2.3点。
原文摘要 · Abstract (English)
Full self-attention is a strong token mixer for PDE surrogates on irregular domains, but its quadratic cost limits its use on high-resolution problems. Efficient latent-attention models such as the Fast Low-rank Attention Routing Engine (FLARE) avoid that cost by routing all N tokens through M << N learned latent queries, but those queries are parameters: once trained, the same learned query templates serve every input. We remove this restriction with FLARE++, a low-rank attention architecture with dynamic token routing. FLARE++ reuses FLARE's own encoder to build its routing queries: learned latent seeds drive one extra encode call that gathers the N input tokens into M input-conditioned queries, and those queries then determine how the same tokens are compressed and redistributed. This preserves FLARE's explicit low-rank factorization and linear O(NM) complexity, and expresses the complete routing operation with standard scaled dot-product attention (SDPA) calls alone. We also provide a multi-GPU context-parallel implementation that shards input tokens across devices without ever gathering the full token sequence on one of them. FLARE++ is competitive across a set of standard PDE surrogate benchmarks, improving on fixed-query FLARE by 24% on average, and it gains 2.3 points of average accuracy on Long Range Arena.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。