通过动态分配计算资源，提升大模型推理时的对齐效果。
Adaptive Blockwise Search: Inference-Time Alignment for Large Language Models
- 根据响应关键位置动态分配搜索计算量，聚焦重要词元。
- 在8个模型上实现超过10%的胜率提升，优于最佳N和微调方法。
- 适合需要高精度对齐但算力有限的应用场景。
大语言模型的对齐仍是关键挑战。推理阶段方法为微调提供了灵活替代，但其均匀的计算开销常导致对齐效果不佳。我们假设，在多数对齐任务中，响应初始词元更为关键。为此，提出AdaSearch，一种新的分块搜索策略，利用采样调度自适应分配固定计算预算，将搜索精力集中于这些关键词元。将AdaSearch应用于序列解码，并引入其树搜索版本AdaBeam。在8个大模型上的全面评估表明，AdaSearch优于强基线Best-of-N和微调方法。具体而言，在无害性生成、可控情感生成及数学推理任务中，胜率相较Best-of-N提升超10%。
原文摘要 · Abstract (English)
LLM alignment remains a critical challenge. Inference-time methods provide a flexible alternative to fine-tuning, but their uniform computational effort often yields suboptimal alignment. We hypothesize that for many alignment tasks, the initial tokens of a response are disproportionately more critical. To leverage this principle, we introduce AdaSearch, a novel blockwise search strategy. It adaptively allocates a fixed computational budget using a sampling schedule, focusing search effort on these critical tokens. We apply AdaSearch to sequential decoding and introduce its tree-search counterpart, AdaBeam. Our comprehensive evaluation across eight LLMs demonstrates that AdaSearch outperforms strong Best-of-N and fine-tuning baselines. Specifically, win-rates improve by over 10% for harmlessness generation, controlled sentiment generation, and for mathematical reasoning tasks relative to Best-of-N.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。