测试大模型处理上百条细粒度约束的能力，发现性能随约束增多大幅下降。
Large-Scale Constraint Generation -- Can LLMs Parse Hundreds of Constraints?
- 构建大规模约束解析任务Words Checker，评估模型处理复杂规则的能力。
- 约束数量增加时，现有方法准确率显著下降，最多损失超10%。
- 提出FoCusNet小模型筛选关键约束，提升大模型专注力与准确率。
近期研究探索了大型语言模型（LLMs）在少量特定任务要求下的约束生成能力。与此相反，本文提出大规模约束生成（LSCG）这一新问题，旨在评估LLMs能否解析大量、细粒度、通用的约束列表。为检验模型在约束数量增加时的表现，我们构建了实际应用场景Words Checker，评估模型特性（如规模、类型）和引导技术（如简单提示、思维链、Best of N）对性能的影响。同时，我们提出FoCusNet，一个小型专用模型，可将原始约束列表精简为更小的相关子集，帮助主模型聚焦关键约束。实验表明，现有方案在约束数量上升时性能显著下降，而FoCusNet带来8-13%的准确率提升。
原文摘要 · Abstract (English)
Recent research has explored the constrained generation capabilities of Large Language Models (LLMs) when explicitly prompted by few task-specific requirements. In contrast, we introduce Large-Scale Constraint Generation (LSCG), a new problem that evaluates whether LLMs can parse a large, fine-grained, generic list of constraints. To examine the LLMs' ability to handle an increasing number constraints, we create a practical instance of LSCG, called Words Checker. In Words Checker, we evaluate the impact of model characteristics (e.g., size, family) and steering techniques (e.g., Simple Prompt, Chain of Thought, Best of N) on performance. We also propose FoCusNet, a small and dedicated model that parses the original list of constraints into a smaller subset, helping the LLM focus on relevant constraints. Experiments reveal that existing solutions suffer a significant performance drop as the number of constraints increases, with FoCusNet showing an 8-13% accuracy boost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。