不修改模型权重，实时调控大模型输出的多目标平衡。
MAVIS: Multi-Objective Alignment via Inference-Time Value-Guided Selection
- 用多个小型价值模型在推理时动态调整输出分布。
- 相比微调单个模型或事后组合，实现更优的多目标权衡。
- 适合需要灵活控制大模型行为的应用场景。
大型语言模型在多样化应用中需平衡多个常冲突的目标（如帮助性、无害性、幽默感）。传统方法需为每种目标或偏好配置微调模型，计算成本高且不灵活。本文提出MAVIS——基于推理时价值引导选择的多目标对齐框架，无需修改基座模型权重即可实现动态行为控制。MAVIS训练一组对应不同目标的小型价值模型，推理时通过用户指定权重组合这些模型，生成倾斜函数以调整基座模型的输出分布。价值模型采用简单迭代算法训练，可实现KL正则化策略的单调优化。实验证明，MAVIS在多目标权衡上优于基线方法，包括针对各目标分别微调后组合，或训练单一偏好条件价值模型进行引导的方法。代码已开源。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed across diverse applications that demand balancing multiple, often conflicting, objectives -- such as helpfulness, harmlessness, or humor. Many traditional methods for aligning outputs to user-specific preferences require fine-tuning models for each objective or for specific preference configurations, which is computationally expensive and inflexible. We introduce \textbf{MAVIS} -- \textit{Multi-Objective Alignment via Inference-Time Value-Guided Selection} -- a lightweight inference-time alignment framework that enables dynamic control over LLM behavior without modifying the base model's weights. MAVIS trains a set of small value models, each corresponding to a distinct objective. At inference time, these value models are combined using user-specified weights to produce a tilting function that adjusts the base model's output distribution toward desired trade-offs. The value models are trained using a simple iterative algorithm that enables monotonic improvement of the KL-regularized policy. We show empirically that MAVIS achieves a superior pareto front compared to baselines which fine-tune per-objective models and combine them post hoc or train a single preference-conditioned value model for guidance. Our code is available at https://github.com/5-Jeremy/MAVIS/tree/main.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。