让大模型在推理时通过文字反馈实时对齐人类偏好，无需重训练。
Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback
- 用文本批评替代数值奖励，迭代优化输出
- 仅几轮就超越预对齐模型，效果显著提升
- 适合需要快速适配人类偏好的实际应用
大型语言模型虽表现优异，但难以在不重训练的情况下快速适应人类偏好。本文提出测试时偏好优化（TPO），在推理阶段实现模型输出与人类偏好的对齐，无需更新模型参数。TPO将奖励信号转化为文本批评，并作为文本奖励迭代改进响应。在涵盖指令遵循、偏好对齐、安全性和数学推理的基准测试中，TPO持续提升与人类偏好的对齐度。值得注意的是，仅经过几轮TPO迭代，原本未对齐的Llama-3.1-70B-SFT模型即可超越预对齐版本Llama-3.1-70B-Instruct。此外，TPO在推理时具有良好的宽度和深度扩展性。案例研究显示，TPO充分利用了大模型理解并响应奖励信号的能力。该方法为测试时偏好优化提供了轻量、高效的实用方案，支持即时对齐。代码已公开于https://github.com/yafuly/TPO。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate impressive performance but lack the flexibility to adapt to human preferences quickly without retraining. In this work, we introduce Test-time Preference Optimization (TPO), a framework that aligns LLM outputs with human preferences during inference, removing the need to update model parameters. Rather than relying on purely numerical rewards, TPO translates reward signals into textual critiques and uses them as textual rewards to iteratively refine its response. Evaluations on benchmarks covering instruction following, preference alignment, safety, and mathematics reveal that TPO progressively improves alignment with human preferences. Notably, after only a few TPO steps, the initially unaligned Llama-3.1-70B-SFT model can surpass the aligned counterpart, Llama-3.1-70B-Instruct. Furthermore, TPO scales efficiently with both the search width and depth during inference. Through case studies, we illustrate how TPO exploits the innate capacity of LLM to interpret and act upon reward signals. Our findings establish TPO as a practical, lightweight alternative for test-time preference optimization, achieving alignment on the fly. Our code is publicly available at https://github.com/yafuly/TPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。