用稀疏注意力让大模型在普通电脑上处理超长文本，还能保持高准确率。
RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
- 通过随机几何稀疏采样，将注意力复杂度从O(N²)降到O(N log N)
- 在32768词长下准确率达75%，超过原版模型的71.88%
- 仅用普通CPU和16-128GB内存即可运行，适合资源有限的研究者
大型语言模型的全量自注意力计算复杂度为O(N²)，限制了长文本处理至65,536个词元，并需昂贵的GPU集群支持。本文提出的无模型依赖架构RIS-Kernel，通过稀疏随机几何采样，将自注意力复杂度降至O(N log N)，且可在消费级内存范围内运行。我们在Qwen2-1.5B-Instruct模型上验证了RIS在两种场景下的表现：在32,768词元的控制实验中，1%密度、70次集成种子的RIS-Stochastic达到75.00%准确率，优于原生密集注意力的71.88%；5%密度、10次种子则匹配该基线。这表明稀疏注意力具有正则化作用：低密度（1%）多种子可滤除序列级噪声，高密度（5%）则引入干扰噪声。在最紧预算下，RIS-Structural在1%密度、10次种子时达68.75%准确率，恢复了相对于零上下文基准（59.38%）75%的上下文优势。在65,536词元时，原生密集注意力触发内存溢出，而RIS相较零上下文基准提升达14.06个百分点（51.56%），经McNemar配对检验（p=0.078<0.10）确认为边际显著。所有实验均在普通未加速CPU服务器（16–128 GB内存）上完成，证明长上下文大模型推理可在标准学术硬件上实现，无需GPU加速。
原文摘要 · Abstract (English)
Full self-attention in large language models scales as O(N^2), which limits long-context document analysis to 65,536 tokens and requires costly GPU clusters. The Reduced Interaction Sampling (RIS) inference engine addresses this constraint as a model-agnostic architecture. Without modifying weights, RIS reduces self-attention complexity to O(N log N) using sparse stochastic geometry that fits within commodity memory limits. We validate RIS on Qwen2-1.5B-Instruct across two regimes. In controlled evaluations at 32,768 tokens (where native dense attention serves as the upper bound), RIS-Stochastic at 1% density and 70 ensemble seeds achieves 75.00% accuracy, outperforming the native dense baseline (71.88%), while RIS-Stochastic at 5% density and 10 seeds matches it (71.88%). This demonstrates that sparse attention acts as a regularizer: low density (1%) over multiple seeds filters out sequence-level noise, whereas higher density (5%) reintroduces distractor noise. Under the tightest budget, RIS-Structural reaches 68.75% accuracy at 1% density with just 10 seeds, recovering 75% of the contextual gap relative to the zero-context floor (59.38%). At 65,536 tokens, where dense attention triggers out-of-memory faults, RIS yields retrieval gains of up to 14.06 percentage points over the zero-context floor (51.56%), which is confirmed as marginally significant under McNemar's paired test (p = 0.078 < 0.10). All evaluations run on commodity, unaccelerated CPU servers (16-128 GB of RAM), demonstrating that long-context LLM inference is feasible on standard academic hardware without GPU acceleration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。