用小模型引导大模型，实现高效测试时对齐。
Kad: A Framework for Proxy-based Test-time Alignment with Knapsack Approximation Deferral
- 通过0-1背包问题建模令牌级延迟决策
- 在任务性能和推测解码速度上均提升
- 适合大规模语言模型的低成本对齐场景
先前研究指出，大型语言模型（LLM）的大部分生成能力在预训练阶段已习得。然而，为满足下游任务需求、风格偏好等目标，仍需进一步对齐。随着模型规模持续扩大，对齐过程的计算开销急剧上升。本文提出一种基于代理模型的测试时对齐框架（Kad），利用小型对齐模型提供指导，实现令牌级渐进式延迟决策。该决策被建模为0-1背包问题，并推导出其原始与对偶近似解。实验表明，该方法在任务表现和推测解码速度上均有显著提升。
原文摘要 · Abstract (English)
Several previous works concluded that the largest part of generation capabilities of large language models (LLM) are learned (early) during pre-training. However, LLMs still require further alignment to adhere to downstream task requirements and stylistic preferences, among other desired properties. As LLMs continue to scale in terms of size, the computational cost of alignment procedures increase prohibitively. In this work, we propose a novel approach to circumvent these costs via proxy-based test-time alignment, i.e. using guidance from a small aligned model. Our approach can be described as a token-specific cascading method, where the token-specific deferral rule is reduced to 0-1 knapsack problem. In this setting, we derive primal and dual approximations of the optimal deferral decision. We experimentally show the benefits of our method both in task performance and speculative decoding speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。