用提升方法优化Transformer，提速增效不改结构
Tricks and Plug-ins for Gradient Boosting with Transformers
- 在Transformer中引入最小二乘提升，动态选重要词元
- 多个文本分类任务上收敛更快、准确率更高
- 无需调参或改架构，适合工程部署与快速实验
Transformer架构主导现代自然语言处理，但常需大量计算资源和复杂的超参数调优。为缓解这些问题，我们提出一种新框架BoostTransformer，通过子网格词元选择和重要性加权采样，将提升原理融入Transformer流程。该方法直接在Transformer管道中引入最小二乘提升目标，实现更高效的训练与更优性能。在多个细粒度文本分类基准上，BoostTransformer展现出更快的收敛速度和更高的准确率，优于标准Transformer，同时显著降低架构搜索开销。
原文摘要 · Abstract (English)
Transformer architectures dominate modern NLP but often demand heavy computational resources and intricate hyperparameter tuning. To mitigate these challenges, we propose a novel framework, BoostTransformer, that augments transformers with boosting principles through subgrid token selection and importance-weighted sampling. Our method incorporates a least square boosting objective directly into the transformer pipeline, enabling more efficient training and improved performance. Across multiple fine-grained text classification benchmarks, BoostTransformer demonstrates both faster convergence and higher accuracy, surpassing standard transformers while minimizing architectural search overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。