用逻辑形式化方法解析对齐模型的损失函数，揭示其内在机制。
Understanding the Logic of Direct Preference Alignment through Logic
- 将偏好对齐损失转化为可形式化的符号程序，建立统一分析框架。
- 揭示不同算法在损失空间中的结构关系与差异本质。
- 为设计新损失函数提供理论依据，适合对齐研究者参考。
近期的直接偏好对齐算法（DPA），如 DPO，展现出将大语言模型与人类偏好对齐的巨大潜力。尽管催生了诸多新变体，但由于缺乏技术与概念框架来理解这些算法的底层语义，理解和开发新的 DPA 损失函数仍具挑战。本文尝试通过将 DPA 损失形式化为离散推理问题来解决这一难题。具体而言：给定一个已有 DPA 损失，能否系统推导出刻画其语义的符号程序？我们提出了针对单模型与参考模型方法的偏好损失符号形式化框架，并识别出若干常用 DPA 变体的符号表达。进一步，该形式化视角揭示了 DPA 损失空间的规模与结构，使我们不仅能严格刻画现有损失之间的关系，还可基于第一性原理系统探索损失空间并推导新损失函数。我们希望该框架与发现能为人类-人工智能对齐研究者提供实用指导。
原文摘要 · Abstract (English)
Recent direct preference alignment algorithms (DPA), such as DPO, have shown great promise in aligning large language models to human preferences. While this has motivated the development of many new variants of the original DPO loss, understanding the differences between these recent proposals, as well as developing new DPA loss functions, remains difficult given the lack of a technical and conceptual framework for reasoning about the underlying semantics of these algorithms. In this paper, we attempt to remedy this by formalizing DPA losses in terms of discrete reasoning problems. Specifically, we ask: Given an existing DPA loss, can we systematically derive a symbolic program that characterizes its semantics? We propose a novel formalism for characterizing preference losses for single model and reference model based approaches, and identify symbolic forms for a number of commonly used DPA variants. Further, we show how this formal view of preference learning sheds new light on both the size and structure of the DPA loss landscape, making it possible to not only rigorously characterize the relationships between recent loss proposals but also to systematically explore the landscape and derive new loss functions from first principles. We hope our framework and findings will help provide useful guidance to those working on human AI alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。