通过空间聚类调度让可变形注意力更适配硬件，显著提升推理速度与能效。
QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attention
- 按空间距离排序查询，预取区域数据形成缓存友好计算流
- 单次遍历完成插值、Softmax、聚合等操作，减少中间结果溢出
- 适合部署在边缘设备或低功耗场景的高效视觉检测任务
可变形注意力虽在目标检测中表现卓越，但因内存访问不规则且算术强度低，难以高效映射到硬件。本文提出QUILL，一种面向调度感知的加速器设计。其核心为基于距离的乱序查询（DOOQ）机制，按空间邻近性排列查询，并通过前瞻预取将区域数据加载至备用缓冲区，形成调度感知的预取循环，实现内存与计算重叠。融合的MSDeformAttn引擎在单次遍历中完成插值、Softmax、聚合及最终投影（W''m），避免中间结果溢出；小张量保留在片上，周边密集层由集成GEMM执行。采用RTL实现并端到端评估显示，相较于RTX 4090，QUILL最高提升7.29倍吞吐量，能效提升47.3倍；相比现有加速器，吞吐量提升3.26–9.82倍，能效提升2.01–6.07倍。结合混合精度量化后，准确率在可变形与稀疏DETR变体上保持与FP32相差≤0.9 AP。通过将稀疏性转化为局部性，再将局部性转化为利用率，实现了持续的端到端加速。
原文摘要 · Abstract (English)
Deformable transformers deliver state-of-the-art detection but map poorly to hardware due to irregular memory access and low arithmetic intensity. We introduce QUILL, a schedule-aware accelerator that turns deformable attention into cache-friendly, single-pass work. At its core, Distance-based Out-of-Order Querying (DOOQ) orders queries by spatial proximity; the look-ahead drives a region prefetch into an alternate buffer--forming a schedule-aware prefetch loop that overlaps memory and compute. A fused MSDeformAttn engine executes interpolation, Softmax, aggregation, and the final projection (W''m) in one pass without spilling intermediates, while small tensors are kept on-chip and surrounding dense layers run on integrated GEMMs. Implemented as RTL and evaluated end-to-end, QUILL achieves up to 7.29x higher throughput and 47.3x better energy efficiency than an RTX 4090, and exceeds prior accelerators by 3.26-9.82x in throughput and 2.01-6.07x in energy efficiency. With mixed-precision quantization, accuracy tracks FP32 within <=0.9 AP across Deformable and Sparse DETR variants. By converting sparsity into locality--and locality into utilization--QUILL delivers consistent, end-to-end speedups.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。