动态稀疏注意力让多示例提示推理更快更省资源。
Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attention
- 用分块稀疏注意力+缓存示例组，实现无需训练的高效检索。
- 单次推理延迟接近微调水平，平均准确率超最优方法95%。
- 适合需要快速响应的多任务部署场景，尤其看重效率的系统。
多示例上下文学习（many-shot ICL）近期展现出替代微调的潜力，其优势在于同一模型可服务多个任务。然而，这将计算负担从训练阶段转移到推理阶段，使多示例 ICL 的实际部署难以落地。若每个推理样本还需定制化检索演示集，成本进一步上升。本文提出动态分块稀疏注意力（Dynamic Block-Sparse Attention），一种无需训练的基于检索的多示例 ICL 框架。通过结合精心设计的分块稀疏注意力机制与缓存示例组的检索策略，该方法在保持平均超过95%最佳方法准确率的同时，实现了与微调相当的单样本推理延迟。我们希望这能推动多示例 ICL 在大规模场景中的应用。
原文摘要 · Abstract (English)
Many-shot in-context learning has recently shown promise as an alternative to finetuning, with the major advantage that the same model can be served for multiple tasks. However, this shifts the computational burden from training-time to inference-time, making deployment of many-shot ICL challenging to justify in-practice. This cost is further increased if a custom demonstration set is retrieved for each inference example. We present Dynamic Block-Sparse Attention, a training-free framework for retrieval-based many-shot in-context learning. By combining carefully designed block-sparse attention and retrieval of cached groups of demonstrations, we achieve comparable per-example latency to finetuning while maintaining on average >95% of the best method's accuracy across strong ICL and finetuning baselines. We hope that this will further enable the deployment of many-shot ICL at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。