通过词元压缩提升小模型推理效率，无需额外训练。
Task-Centric Acceleration of Small-Language Models
- 迭代扩展词表并微调，增强小模型输出能力。
- 推理时用任务语料构建轻量草案模型，提速30%以上。
- 适合低多样性生成场景，如代码补全、摘要生成。
小语言模型（SLMs）已成为特定任务中高效替代大模型的选择。但在高吞吐、低延迟场景下，效率至关重要。本文提出TASC（任务自适应序列压缩），包含两种应用场景：在微调阶段，提出TASC-ft，通过迭代将高频输出n-gram加入词表，并微调模型以利用扩展词表；在推理阶段，提出TASC-spec，一种无需训练的轻量级推测解码方法，从任务输出语料中构建n-gram草案模型，融合任务与上下文n-gram信息。该方法避免了草案-目标词表对齐约束，且无需额外训练。我们在多个低输出多样性生成任务上验证其有效性，结果表明两种方法在保持任务性能的同时显著提升推理效率。
原文摘要 · Abstract (English)
Small language models (SLMs) have emerged as efficient alternatives to large language models for task-specific applications. However, they are often employed in high-volume, low-latency settings, where efficiency is crucial. We propose TASC, Task-Adaptive Sequence Compression, a framework for SLM acceleration comprising two use-cases: When performing SLM fine-tuning, we propose TASC-ft, which iteratively enriches the tokenizer vocabulary with high-frequency output n-grams and then fine-tunes the model to utilize the expanded vocabulary. Next, we propose an inference-time method, termed TASC-spec. TASC-spec is a lightweight, training-free speculative decoding method that constructs an n-gram draft model from the task's output corpus, mixing task and context n-gram information.TASC-spec avoids any additional training, while bypassing draft-target vocabulary alignment constraints. We demonstrate the effectiveness of both methods across multiple low output-variability generation tasks. Our methods show consistent improvements in inference efficiency while maintaining task performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。