用轻量方法在低算力下实现大模型对齐，兼顾效果与成本。
Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models
- 用启发式奖励模型+两阶段过滤，减少推理次数
- 在少至1~2次响应下仍保持对齐效果
- 适合资源受限场景的个性化大模型部署
将大语言模型与用户偏好对齐对实际应用至关重要，但通常需要昂贵的微调或高成本推理，导致对齐质量与计算开销之间难以平衡。现有推理时对齐方法往往忽略这一权衡，仅关注优化策略的性能表现。本文提出HIA（Heuristic-Guided Inference-time Alignment），一种无需调参、兼容黑盒的方案，结合轻量级提示优化器、启发式奖励模型和两阶段过滤机制，在不增加计算负担的前提下减少推理调用次数，同时保持对齐质量。在真实提示数据集HelpSteer和ComPRed上，HIA在相同推理预算下，优于最佳N采样、束搜索和贪婪搜索基线，在多目标、目标条件任务中表现更优。此外，即使在极低推理预算（仅需1~2次响应）下，HIA仍具有效性，为可扩展、个性化的大型语言模型部署提供实用解决方案。
原文摘要 · Abstract (English)
Aligning LLMs with user preferences is crucial for real-world use but often requires costly fine-tuning or expensive inference, forcing trade-offs between alignment quality and computational cost. Existing inference-time methods typically ignore this balance, focusing solely on the optimized policy's performance. We propose HIA (Heuristic-Guided Inference-time Alignment), a tuning-free, black-box-compatible approach that uses a lightweight prompt optimizer, heuristic reward models, and two-stage filtering to reduce inference calls while preserving alignment quality. On real-world prompt datasets, HelpSteer and ComPRed, HIA outperforms best-of-N sampling, beam search, and greedy search baselines in multi-objective, goal-conditioned tasks under the same inference budget. We also find that HIA is effective under low-inference budgets with as little as one or two response queries, offering a practical solution for scalable, personalized LLM deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。