用网格化宪法生成数据，提升强化学习对齐的数据效率。
A Constitution-Grid Instrument for Data-Efficient RL Alignment (C-Guard)
- 构建宪法网格自动生成训练数据，动态调整学习区域。
- 减少无效数据投入，使学习效果从0.733提升至0.80。
- 适合关注安全对齐与数据效率的RL研究者使用。
强化学习对齐中普遍存在目标冲突问题，高效训练极具挑战。训练安全防护机制时需同时优化两个矛盾目标：识别真实危害，不误拒良性请求。我们发现，过度拒绝使性能从22.4%降至12.8%，而对抗攻击下的拒绝不足则使安全评分从0.27恶化至0.33。为此提出C-Guard——一种基于宪法网格的训练数据生成工具，以及每单元学习度量（C-LIM），用于决策各单元是否剪枝、加密、修改或扩展。C-LIM可在训练前识别无效数据区域：187条无针对性样本未带来任何收益，本方法将其学习影响提升至0.80（原为0.733）。代码与宪法均已开源。
原文摘要 · Abstract (English)
Conflicting objectives are general in RL alignment, and training on them data-efficiently is hard. Training a safety guard with RL means optimizing two objectives that conflict: catch real harm, and do not refuse benign prompts. Our finding is that over-refusal improves 22.4% to 12.8%, while under-refusal on adversarial attacks silently worsens 0.27 to 0.33. We present C-Guard, a constitution-grid instrument that generates the RL training data, and C-LIM, a per-cell learnability score that decides each cell's move: prune, densify, amend, expand. C-LIM flags the dead-weight data region before any training budget is spent: 187 untargeted rows had bought zero gain, and our method lifts the same region's learning impact 0.733 to 0.80. Code and the constitution are open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。