让语言模型学会自然语言安全约束，提升真实场景下的可靠性。
Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agents
- 从正负样例中学习自然语言约束，作为对齐的首要步骤。
- 在域迁移下仍保持低违规率，使用精简模型时实现零违规。
- 适合构建安全关键型语言应用，尤其关注鲁棒性与泛化能力。
通用对齐是将大语言模型（LLMs）安全部署于真实世界NLP应用的核心挑战。当前对齐方法，如基于人类反馈的强化学习（RLHF），因依赖隐式、事后偏好，常无法保证训练分布外的约束满足。受先构建数据再微调范式的启发，我们提出一种新的安全语言对齐框架，通过正负示范学习自然语言约束作为首要步骤。该框架可推断任务特定奖励函数与潜在约束函数，促进对新安全需求的适应及在域迁移和对抗输入下的鲁棒泛化。我们在文本导航环境中形式化该框架，验证其对动态危险区域的安全适应能力。实验表明，在域迁移时遵循安全路径的违规更少；通过将学习到的约束应用于蒸馏后的BERT模型进行微调，实现零违规。本工作为构建安全关键且更具泛化能力的实用化语言模型提供了可行路径。
原文摘要 · Abstract (English)
Generalizable alignment is a core challenge for deploying Large Language Models (LLMs) safely in real-world NLP applications. Current alignment methods, including Reinforcement Learning from Human Feedback (RLHF), often fail to guarantee constraint satisfaction outside their training distribution due to their reliance on implicit, post-hoc preferences. Inspired by a paradigm shift to first curate data before tuning, we introduce a new framework for safe language alignment that learns natural language constraints from positive and negative demonstrations as a primary step. From inferring both a task-specific reward function and latent constraint functions, our approach fosters adaptation to novel safety requirements and robust generalization under domain shifts and adversarial inputs. We formalize the framework within a Constrained Markov Decision Process (CMDP) and validate it via a text-based navigation environment, demonstrating safe adaptation to changing danger zones. Our experiments show fewer violations upon domain shift when following a safe navigation path, and we achieve zero violations by applying learned constraints to a distilled BERT model as a fine-tuning technique. This work offers a promising path toward building safety-critical and more generalizable LLMs for practical NLP settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。