通过递归与高效注意力结合,实现大模型推理时的高效深度扩展。
Universal YOCO for Efficient Depth Scaling
- 采用共享参数的递归自解码器,仅在浅层注意力层迭代。
- 保持恒定全局键值缓存,前填充计算线性增长。
- 适合需要长上下文且追求推理效率的场景。
测试时扩展显著提升了大语言模型的推理与代理能力。然而,标准Transformer在推理时扩展计算效率低下,因传统循环策略带来高开销,且键值缓存随模型深度增加而膨胀。本文提出通用YOCO(YOCO-U),将YOCO解码器-解码器架构与递归计算相结合,产生协同增益。基于YOCO框架,YOCO-U实现通用自解码器,通过参数共享执行多轮迭代,但仅限于浅层、高效的注意力层。该设计实现了能力-效率的优越权衡,单靠YOCO或递归均无法达成。YOCO架构提供恒定全局键值缓存和线性预填充,部分递归以低开销提升表征深度。联合效果提升了令牌利用率与可扩展性,同时维持高效推理。实证结果表明,YOCO-U在通用及长上下文基准测试中表现优异,验证了高效注意力架构与递归计算融合是可扩展大模型的有前景方向。
原文摘要 · Abstract (English)
The rise of test-time scaling has remarkably boosted the reasoning and agentic proficiency of Large Language Models (LLMs). Yet, standard Transformers struggle to scale inference-time compute efficiently, as conventional looping strategies suffer from high computational overhead and a KV cache that inflates alongside model depth. We present Universal YOCO (YOCO-U), which combines the YOCO decoder-decoder architecture with recursive computation to achieve a synergistic effect greater than either alone. Built on the YOCO framework, YOCO-U implements a Universal Self-Decoder that performs multiple iterations via parameter sharing, while confining the iterative process to shallow, efficient-attention layers. This combination yields a favorable capability-efficiency tradeoff that neither YOCO nor recursion achieves independently. The YOCO architecture provides a constant global KV cache and linear pre-filling, while partial recursion enhances representational depth with limited overhead. Together, YOCO-U improves token utility and scaling behavior while maintaining efficient inference. Empirical results confirm that YOCO-U remains highly competitive in general and long-context benchmarks, demonstrating that the integration of efficient-attention architectures and recursive computation is a promising direction for scalable LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。