提出修正机制，让低效稀疏模型在推理任务中恢复性能。
Sirius: Contextual Sparsity with Correction for Efficient LLMs
- 通过少量令牌修正，恢复稀疏模型的推理能力
- 在8个复杂任务上使80亿和700亿模型延迟降低20%至35%
- 适合追求高效推理且需保持推理质量的研究者
随着大语言模型的发展，推理效率日益重要。上下文稀疏（CS）因其无需训练且压缩率高而受到关注，但在复杂生成任务中，其在推理、推断和知识型任务上表现显著下降。尽管端到端准确率存在差距，我们发现稀疏模型仍具备通用求解逻辑，仅需少量令牌修正即可恢复原模型性能。本文提出Sirius，一种高效的修正机制，在6个模型、8个困难推理、数学与编码任务上均实现性能显著恢复，同时保持高效性。我们还实现了系统级部署，验证了8B模型片上推理延迟降低约20%，70B模型离线推理降低35%。代码已开源：https://github.com/Infini-AI-Lab/Sirius.git。
原文摘要 · Abstract (English)
With the blossom of large language models (LLMs), inference efficiency becomes increasingly important. Various approximation methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without quality degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, CS significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models often share general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces Sirius, an efficient correction mechanism, which significantly recovers CS models quality on reasoning tasks while maintaining its efficiency gain. Sirius is evaluated on 6 models with 8 difficult generation tasks in reasoning, math, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for Sirius and show that Sirius achieves roughly 20% reduction in latency for 8B model on-chip and 35% reduction for 70B model offloading. We open-source our implementation of Sirius at https://github.com/Infini-AI-Lab/Sirius.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。