用数学约束替代经验调参，让大模型按需求精准对齐
L3Ms -- Lagrange Large Language Models
- 将微调与对齐建模为带约束的优化问题
- 通过对数障碍函数实现无启发式依赖的约束满足
- 适用于需定制化对齐的多种应用场景
监督微调（SFT）和大语言模型（LLM）的对齐是提升用户体验的关键步骤。然而，合适的对齐方式本质上依赖具体应用，现有方法常依赖启发式选择驱动优化。本文将SFT与对齐形式化为一个约束优化问题：在任务上微调LLM的同时，需满足特定应用要求，且不依赖启发式。为此，我们提出拉格朗日大语言模型（L3Ms），采用对数障碍函数强制约束。该方法可在不同应用间灵活定制，避免启发式过程。实验表明，L3Ms在多种应用中均能实现针对性对齐，展现出卓越的通用性与有效性。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) and alignment of large language models (LLMs) are key steps in providing a good user experience. However, the concept of an appropriate alignment is inherently application-dependent, and current methods often rely on heuristic choices to drive optimization. In this work, we formulate SFT and alignment as a constrained optimization problem: the LLM is fine-tuned on a task while being required to meet application-specific requirements, without resorting to heuristics. To solve this, we propose Lagrange Large Language Models (L3Ms), which employ logarithmic barriers to enforce the constraints. This approach allows for the customization of L3Ms across diverse applications while avoiding heuristic-driven processes. We experimentally demonstrate the versatility and efficacy of L3Ms in achieving tailored alignments for various applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。