构建可精确验证的指令遵循评估框架，提升大模型可控性研究可信度。
LexInstructEval: Lexical Instruction Following Evaluation for Large Language Models
- 基于规则语法拆解复杂指令为三元组，实现结构化表达
- 通过人工参与的多阶段流程生成多样数据集，保证覆盖度
- 提供开源工具链，支持透明、可复现的自动化评测
大语言模型精准执行复杂细粒度词汇指令的能力是其可用性和可控性的核心。然而，评估这一能力仍面临重大挑战：现有方法或依赖主观且成本高昂的人工评价，或采用存在固有偏见和不可靠性的自动化大模型评分系统；而现有的程序化基准虽客观，却难以测试精细、组合式的约束条件。为此，我们提出LexInstructEval，一个用于细粒度词汇指令遵循的新基准与评估框架。该框架基于形式化的规则语法，将复杂指令分解为标准的<操作, 关系, 值>三元组，通过多阶段、人工参与的流水线系统生成多样化数据集，并借助透明的程序化引擎实现客观验证。我们已发布数据集及开源评估工具，以促进大模型可控性与可靠性研究的深入发展。
原文摘要 · Abstract (English)
The ability of Large Language Models (LLMs) to precisely follow complex and fine-grained lexical instructions is a cornerstone of their utility and controllability. However, evaluating this capability remains a significant challenge. Current methods either rely on subjective and costly human evaluation or on automated LLM-as-a-judge systems, which suffer from inherent biases and unreliability. Existing programmatic benchmarks, while objective, often lack the expressiveness to test intricate, compositional constraints at a granular level. To address these limitations, we introduce LexInstructEval, a new benchmark and evaluation framework for fine-grained lexical instruction following. Our framework is built upon a formal, rule-based grammar that deconstructs complex instructions into a canonical <Procedure, Relation, Value> triplet. This grammar enables the systematic generation of a diverse dataset through a multi-stage, human-in-the-loop pipeline and facilitates objective verification via a transparent, programmatic engine. We release our dataset and open-source evaluation tools to facilitate further research into the controllability and reliability of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。