让用户用自然语言定义硬性规则和柔性偏好,提升大模型规划的可控性。
U-Define: Designing User Workflows for Hard and Soft Constraints in LLM-Based Planning

- 将约束分为必须遵守的硬规则和可灵活处理的软偏好
- 用形式化验证和大模型评分分别检查硬软约束的满足情况
- 用户实测表明该方法显著提升任务满意度与执行效果
大语言模型在用户任务规划中的应用日益广泛,但其黑箱特性限制了用户对结果可靠性和控制力的把握。现有系统虽引入验证机制,但用户难以有效运用刚性约束表达意图或应对现实变化。例如,纯硬约束过于僵化,数值灵活性权重又令用户困惑。本文研究如何通过交互流程更好支持用户设定约束以引导大模型生成计划,探究将严格度抽象为高阶类型(硬/软)并搭配不同验证方式是否有助于更可靠地表达和对齐意图。我们提出U-Define系统,允许用户以自然语言定义约束,并分类为必须不违反的硬规则或可接受灵活性的软偏好。系统采用互补验证策略:硬约束使用形式化模型检查,软约束由大模型作为评判者评估。通过技术评估及面向普通用户与专家的用户研究,结果表明用户定义的约束类型显著提升感知有用性、性能与满意度,同时保持易用性。研究为设计灵活而可靠的约束工作流提供了重要启示。
原文摘要 · Abstract (English)
LLMs are increasingly used for end-user task planning, yet their black-box nature limits users' ability to ensure reliability and control. While recent systems incorporate verification techniques, it remains unclear how users can effectively apply such rigid constraints to represent intent or adapt to real-world variability. For example, prior work finds that hard-only constraints are too rigid, and numeric flexibility weights confuse users. We investigate how interaction workflows can better support users in applying constraints to guide LLM-generated plans, examining whether abstracting strictness into high-level types (i.e., hard and soft) paired with distinct verification mechanisms helps users more reliably express and align intent. We present U-Define, a system that lets users define constraints in natural language and categorize them as either hard rules that must not be violated or soft preferences that allow flexibility. U-Define verifies these types through complementary methods: formal model checking for hard constraints and LLM-as-judge evaluation for soft ones. Through a technical evaluation and user studies with general and expert participants, we find that user-defined constraint types improve perceived usefulness, performance, and satisfaction while maintaining usability. These findings provide insights for designing flexible yet reliable constraint-based workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。