优化大模型推理速度，通过渐进压缩提升效率
Revisiting Funnel Transformers for Modern LLM Architectures with Comprehensive Ablations in Training and Inference Configurations
- 在Gemma2模型中系统测试不同压缩层和恢复策略
- 选择合适压缩层可降低44%延迟，避免严重性能下降
- 适合关注大模型部署效率与精度平衡的研究者
基于Transformer的大语言模型因计算成本高而发展迅速，早期优化方法未必适用于现代模型。本文以Dai和Le（2020）提出的漏斗Transformer为基础，系统评估其在现代Gemma2架构中的表现。通过对比标准预训练与漏斗感知预训练、漏斗感知微调以及序列恢复操作类型，发现漏斗结构会在深层网络中引入信息瓶颈，尤其在大型模型（如Gemma 7B）中导致显著性能损失。但合理选择漏斗层并采用有效恢复策略，可大幅缓解损失，实现最高达44%的延迟降低。研究揭示了计算效率与模型准确率之间的关键权衡，为大规模自然语言应用中部署漏斗型架构提供实用指导。
原文摘要 · Abstract (English)
Transformer-based Large Language Models, which suffer from high computational costs, advance so quickly that techniques proposed to streamline earlier iterations are not guaranteed to benefit more modern models. Building upon the Funnel Transformer proposed by Dai and Le (2020), which progressively compresses intermediate representations, we investigate the impact of funneling in contemporary Gemma2 Transformer architectures. We systematically evaluate various funnel configurations and recovery methods, comparing: (1) standard pretraining to funnel-aware pretraining strategies, (2) the impact of funnel-aware fine-tuning, and (3) the type of sequence recovery operation. Our results demonstrate that funneling creates information bottlenecks that propagate through deeper network layers, particularly in larger models (e.g., Gemma 7B), leading to at times unmanageable performance lost. However, carefully selecting the funneling layer and employing effective recovery strategies, can substantially mitigate performance losses, achieving up to a 44\% reduction in latency. Our findings highlight key trade-offs between computational efficiency and model accuracy, providing practical guidance for deploying funnel-based approaches in large-scale natural language applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。