提出新测试时扩展规律,证明大模型更高效,稀疏注意力是关键。
Kinetics: Rethinking Test-Time Scaling Laws
- 从计算与内存访问双重成本出发,重构测试时扩展规律。
- 稀疏注意力使小成本下准确率提升超60点,大成本下仍增5点以上。
- 适合关注推理效率、长文本生成与资源优化的研究者。
我们从实际效率角度重新审视测试时扩展规律,发现小模型的有效性被严重高估。以往工作基于计算最优性,忽视了推理策略(如Best-of-$N$、长思维链)带来的关键内存访问瓶颈。通过对0.6B至32B参数模型的全面分析,我们提出新的Kinetics扩展规律,该规律同时考虑计算与内存访问成本。结果显示,测试时计算资源在超过阈值的模型上使用更有效;核心原因是,在测试时推理中,注意力开销而非参数量成为主导因素。受此启发,我们提出以稀疏注意力为核心的新型扩展范式,降低每标记成本,支持更长生成和更多并行样本。实验表明,稀疏注意力模型持续优于稠密模型,在AIME问题求解任务上,低预算场景准确率提升超60点,高预算场景亦提升超5点,涵盖对当前先进MoE模型的评估。结果表明,稀疏注意力对实现测试时扩展潜力至关重要,且随着投入算力增加,其重要性愈发凸显——与训练不同,测试时精度尚未饱和,可通过增加生成量持续提升。代码已开源:https://github.com/Infini-AI-Lab/Kinetics。
原文摘要 · Abstract (English)
We rethink test-time scaling laws from a practical efficiency perspective, revealing that the effectiveness of smaller models is significantly overestimated. Prior work, grounded in compute-optimality, overlooks critical memory access bottlenecks introduced by inference-time strategies (e.g., Best-of-$N$, long CoTs). Our holistic analysis, spanning models from 0.6B to 32B parameters, reveals a new Kinetics Scaling Law that better guides resource allocation by incorporating both computation and memory access costs. Kinetics Scaling Law suggests that test-time compute is more effective when used on models above a threshold than smaller ones. A key reason is that in TTS, attention, rather than parameter count, emerges as the dominant cost factor. Motivated by this, we propose a new scaling paradigm centered on sparse attention, which lowers per-token cost and enables longer generations and more parallel samples within the same resource budget. Empirically, we show that sparse attention models consistently outperform dense counterparts, achieving over 60 points gains in low-cost regimes and over 5 points gains in high-cost regimes for problem-solving accuracy on AIME, encompassing evaluations on state-of-the-art MoEs. These results suggest that sparse attention is essential and increasingly important with more computing invested, for realizing the full potential of test-time scaling where, unlike training, accuracy has yet to saturate as a function of computation, and continues to improve through increased generation. The code is available at https://github.com/Infini-AI-Lab/Kinetics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。