通过预输出空间重要性采样，实现低成本高效语言模型对齐。
Test-time reward-guided alignment of language models by importance sampling on pre-logit space
- 在预输出层施加高斯扰动，优化扰动均值以最大化期望奖励。
- 用较少样本获得更高奖励，优于最佳n选一等现有方法。
- 适合需要快速调优且无资源微调的场景，如部署阶段调整。
大型语言模型（LLM）的测试时对齐因微调成本过高而受到关注。本文提出一种基于随机控制输入的采样式模型预测控制的新方法——预输出空间自适应重要性采样（AISP）。AISP 在网络倒数第二层的预输出（pre-logits）上施加高斯扰动，通过重要性采样优化扰动均值以最大化期望奖励。实验表明，该方法在使用更少样本的情况下，奖励表现优于最佳n选一采样，并超越其他基于奖励的测试时对齐方法。
原文摘要 · Abstract (English)
Test-time alignment of large language models (LLMs) attracts attention because fine-tuning of LLMs requires high computational costs. In this paper, we propose a new test-time reward-guided alignment method called adaptive importance sampling on pre-logits (AISP) on the basis of the sampling-based model predictive control with the stochastic control input. AISP applies the Gaussian perturbation into pre-logits, which are outputs of the penultimate layer, so as to maximize expected rewards with respect to the mean of the perturbation. We demonstrate that the optimal mean is obtained by importance sampling with sampled rewards. AISP outperforms best-of-n sampling in terms of rewards over the number of used samples and achieves higher rewards than other reward-based test-time alignment methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。