让专家协作设计大模型行为规则，实时测试并快速迭代。
PolicyPad: Collaborative Prototyping of LLM Policies
- 结合用户体验设计方法，支持专家实时协同起草政策。
- 在心理与法律领域实验中，显著提升协作效率与反馈速度。
- 适合需要专家参与的高风险AI系统对齐与安全设计。
随着大模型在心理健康等高风险领域应用增多，领域专家日益参与其行为规范制定。通过对19场历时15周、涉及9位专家的政策制定工作坊观察，我们发现现有流程缺乏对快速实验、反馈与迭代的支持。为此提出PolicyPad，一个交互式系统，借鉴用户体验原型设计中的启发式评估与故事板方法，使政策设计者可实时协作撰写政策，并独立通过使用场景测试政策引导下的模型行为。在8组共22位心理健康与法律领域专家的工作坊中评估显示，PolicyPad提升了协作质量，实现了紧密的反馈循环，并催生了新的政策提案。本研究为专家参与的大模型对齐与安全发展提供了可行路径。
原文摘要 · Abstract (English)
As LLMs gain adoption in high-stakes domains like mental health, domain experts are increasingly consulted to provide input into policies governing their behavior. From an observation of 19 policymaking workshops with 9 experts over 15 weeks, we identified opportunities to better support rapid experimentation, feedback, and iteration for collaborative policy design processes. We present PolicyPad, an interactive system that facilitates the emerging practice of LLM policy prototyping by drawing from established UX prototyping practices, including heuristic evaluation and storyboarding. Using PolicyPad, policy designers can collaborate on drafting a policy in real time while independently testing policy-informed model behavior with usage scenarios. We evaluate PolicyPad through workshops with 8 groups of 22 domain experts in mental health and law, finding that PolicyPad enhanced collaborative dynamics during policy design, enabled tight feedback loops, and led to novel policy contributions. Overall, our work paves expert-informed paths for advancing AI alignment and safety.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。