让强模型指导弱模型，提升AI对齐能力。
Explanation, Debate, Align: A Weak-to-Strong Framework for Language Model Generalization
- 强模型通过引导弱模型学习，实现能力迁移。
- 无需大量训练数据，即可提升弱模型性能。
- 适用于多智能体协作与人机团队场景。
人工智能系统的快速进步使AI对齐问题成为研究重点，尤其在复杂决策和任务执行中。随着系统在复杂问题上超越人类表现，确保其与人类价值观、意图和伦理准则对齐变得至关重要。基于先前关于解释生成的人机对齐工作，本文探讨了多智能体系统及人机团队中的更复杂动态。提出一种基于弱到强泛化的新方法，通过强模型促进弱模型的改进，弥合解释生成与模型对齐之间的差距。该方法形式化为一个引导函数，使先进模型的能力可在不直接访问大规模训练数据的情况下迁移到较弱模型。实验结果表明，这种基于引导的方法不仅提升了模型性能，还揭示了模型对齐的本质，并展示了对AI系统进行可扩展监督的潜力。
原文摘要 · Abstract (English)
The rapid advancement of artificial intelligence systems has brought the challenge of AI alignment to the forefront of research, particularly in complex decision-making and task execution. As these systems surpass human-level performance in sophisticated problems, ensuring their alignment with human values, intentions, and ethical guidelines becomes crucial. Building on previous work in explanation generation for human-agent alignment, we address the more complex dynamics of multi-agent systems and human-AI teams. This paper introduces a novel approach to model alignment through weak-to-strong generalization in the context of language models. We present a framework where a strong model facilitates the improvement of a weaker model, bridging the gap between explanation generation and model alignment. Our method, formalized as a facilitation function, allows for the transfer of capabilities from advanced models to less capable ones without direct access to extensive training data. Our results suggest that this facilitation-based approach not only enhances model performance but also provides insights into the nature of model alignment and the potential for scalable oversight of AI systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。