分阶段优化大模型推理，提速降耗且不损效果
OverFill: Two-Stage Models for Efficient Language Model Decoding
- 前向阶段用全模型并行处理，生成阶段用压缩模型逐字输出
- 3B→1B配置比1B压缩模型性能提升83.2%，8B→3B提升79.2%
- 训练数据少但性能媲美原生训练模型，适合部署优化场景
大语言模型在各类任务中表现优异，但高推理成本制约其部署。模型推理包含计算密集的prefill阶段和内存密集的decode阶段，后者在长序列下主导延迟。现有解码器仅模型统一处理两阶段，未考虑其差异性。我们提出OverFill，将两阶段解耦：先用完整模型并行处理系统与用户输入，再切换至稀疏化压缩模型逐个生成词元。通过在prefill阶段增加计算资源，显著提升生成质量且延迟几乎不变。3B→1B配置在标准基准上较1B压缩模型性能提升83.2%，8B→3B配置较3B压缩模型平均提升79.2%。OverFill性能与同规模从头训练模型相当，但所需训练数据大幅减少。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) excel across diverse tasks but face significant deployment challenges due to high inference costs. LLM inference comprises prefill (compute-bound) and decode (memory-bound) stages, with decode dominating latency particularly for long sequences. Current decoder-only models handle both stages uniformly, despite their distinct computational profiles. We propose OverFill, which decouples these stages to optimize accuracy-efficiency tradeoffs. OverFill begins with a full model for prefill, processing system and user inputs in parallel. It then switches to a dense pruned model, while generating tokens sequentially. Leveraging more compute during prefill, OverFill improves generation quality with minimal latency overhead. Our 3B-to-1B OverFill configuration outperforms 1B pruned models by 83.2%, while the 8B-to-3B configuration improves over 3B pruned models by 79.2% on average across standard benchmarks. OverFill matches the performance of same-sized models trained from scratch, while using significantly less training data. Our code is available at https://github.com/friendshipkim/overfill.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。