通过词元级好坏判断，低成本提升大模型对齐效果。
Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models
- 将句子对齐拆解为词元级二分类，用小网络判断每个词元是否可接受
- 在7个大模型上验证，对齐效果显著提升且计算开销降低
- 适合需要高效对齐但算力有限的场景，开源代码和模型可用
随着大语言模型的快速发展，使其与人类偏好和价值观对齐对于确保其伦理性和安全性至关重要。然而，现有的对齐方法如基于强化学习的人类反馈（RLHF）或直接偏好优化（DPO）通常需要在拥有数十亿参数的大模型上进行微调，导致巨大的计算成本和效率问题。为此，我们提出一种独立于语言模型的微粒度词元级接受-拒绝对齐方法（MARA）。该方法将句子级别的偏好学习分解为词元级别的二分类任务，使用一个小型三层全连接网络判断候选词元是否应被“接受”或“拒绝”作为响应的一部分。在七个不同大模型和三个开源数据集上的大量实验表明，MARA在显著提升对齐性能的同时大幅降低了计算成本。源代码和实现细节已公开于https://github.com/IAAR-Shanghai/MARA，训练好的模型也已发布于https://huggingface.co/IAAR-Shanghai/MARA_AGENTS。
原文摘要 · Abstract (English)
With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。