LoZA让长文本模型提速，100万词输入仅需少量算力
Efficient Context Scaling with LongCat ZigZag Attention
- 用稀疏注意力机制改造现有模型，节省计算资源
- 在百万级上下文下推理速度显著提升，支持长程思考
- 适合需要长期推理的智能体系统，如工具调用和检索增强
我们提出LongCat ZigZag Attention（LoZA），一种稀疏注意力方案，可将任意全注意力模型转化为低计算开销的稀疏版本。在长上下文场景中，LoZA在预填充密集（如检索增强生成）和解码密集（如工具集成推理）任务中均实现显著加速。具体而言，通过在训练中期应用LoZA于LongCat-Flash，我们构建了LongCat-Flash-Exp这一长上下文基础模型，可高效处理高达100万词的输入，支持高效的长期推理与长时序智能体能力。
原文摘要 · Abstract (English)
We introduce LongCat ZigZag Attention (LoZA), which is a sparse attention scheme designed to transform any existing full-attention models into sparse versions with rather limited compute budget. In long-context scenarios, LoZA can achieve significant speed-ups both for prefill-intensive (e.g., retrieval-augmented generation) and decode-intensive (e.g., tool-integrated reasoning) cases. Specifically, by applying LoZA to LongCat-Flash during mid-training, we serve LongCat-Flash-Exp as a long-context foundation model that can swiftly process up to 1 million tokens, enabling efficient long-term reasoning and long-horizon agentic capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。