PHOTON用分层垂直扫描替代传统逐词生成，显著提升长文本推理速度与内存效率。
PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation
- 构建分层潜流结构，上下文压缩与并行解码分离，减少冗余计算。
- 解码时仅更新最粗粒度潜流，使每单位内存吞吐量提升千倍。
- 适合长文本生成、多查询场景，尤其在高延迟和内存受限环境下表现突出。
Transformer采用水平的逐词扫描机制，在每个生成步骤中需访问不断增长的词元状态序列，导致预填充延迟增加，且长上下文解码受内存瓶颈制约，因为KV缓存的读写主导了推理时间。本文提出平行分层操作的自回归网络（PHOTON），将水平扫描改为垂直、多分辨率的上下文扫描。PHOTON维护一组分层潜流：自底向上的编码器将词元压缩为低速率上下文状态，而轻量级自顶向下的解码器并行重构细粒度词元表示。进一步引入递归生成机制，仅更新最粗粒度潜流，避免重新编码底层状态。实验表明，相较于主流Transformer模型，PHOTON在吞吐量-质量权衡上表现更优，尤其在长上下文与多查询任务中。其显著降低解码阶段的KV缓存流量，实现每单位内存高达10³倍的吞吐提升。
原文摘要 · Abstract (English)
Transformers operate as horizontal token-by-token scanners; at each generation step, attending to an ever-growing sequence of token-level states. This access pattern increases prefill latency and makes long-context decoding more memory-bound, as KV-cache reads and writes dominate inference time over arithmetic operations. We propose Parallel Hierarchical Operation for TOp-down Networks (PHOTON), a hierarchical autoregressive model that replaces horizontal scanning with vertical, multi-resolution context scanning. PHOTON maintains a hierarchy of latent streams: a bottom-up encoder compresses tokens into low-rate contextual states, while lightweight top-down decoders reconstruct fine-grained token representations in parallel. We further introduce recursive generation that updates only the coarsest latent stream and eliminates bottom-up re-encoding. Experimental results show that PHOTON is superior to competitive Transformer-based language models regarding the throughput-quality trade-off, providing advantages in long-context and multi-query tasks. In particular, this reduces decode-time KV-cache traffic, yielding up to $10^{3}\times$ higher throughput per unit memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。