用虚拟令牌提升大模型推理能力，无需大量训练。
Enhancing Latent Computation in Transformers with Latent Tokens
- 引入虚拟令牌，通过注意力机制引导解码过程。
- 在分布外场景下性能显著优于基线模型。
- 轻量高效，可灵活用于已有大模型系统。
为增强大语言模型（LLM）性能，本文提出一种轻量级方法——潜在令牌（latent tokens）。这些虚拟令牌虽在自然语言中无意义，但可通过 Transformer 的注意力机制引导自回归解码过程。潜在令牌可无缝集成至预训练的 Transformer 模型中，以参数高效的方式进行训练，并在推理阶段灵活使用，对现有架构复杂度影响极小。我们提出了关于潜在令牌作用机制的若干假设，并设计了合成任务进行验证。实验结果表明，该方法在分布外泛化场景中表现显著优于基线模型，展现出提升大模型适应性的潜力。
原文摘要 · Abstract (English)
Augmenting large language models (LLMs) with auxiliary tokens has emerged as a promising strategy for enhancing model performance. In this work, we introduce a lightweight method termed latent tokens; these are dummy tokens that may be non-interpretable in natural language but steer the autoregressive decoding process of a Transformer-based LLM via the attention mechanism. The proposed latent tokens can be seamlessly integrated with a pre-trained Transformer, trained in a parameter-efficient manner, and applied flexibly at inference time, while adding minimal complexity overhead to the existing infrastructure of standard Transformers. We propose several hypotheses about the underlying mechanisms of latent tokens and design synthetic tasks accordingly to verify them. Numerical results confirm that the proposed method noticeably outperforms the baselines, particularly in the out-of-distribution generalization scenarios, highlighting its potential in improving the adaptability of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。