用类型理论自动评估大模型指令遵循能力，更贴近人类判断。
DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following
- 双模型协作分解指令为带类型的逻辑命题
- 对不同类型命题采用不同评估标准，准确率达90.38%
- 适合评估复杂多轮对话中的指令遵循行为
评估大语言模型的指令遵循能力需要将指令拆解为可验证的要求并评估其满足程度——当前方法依赖人工标注且标准统一，与人类判断模式不符。我们提出DIALEVAL，一种基于类型理论的框架，利用双大模型代理自动将指令分解为带类型的谓词，并实现类型相关的满足语义。该框架在自动提取过程中强制执行形式化的原子性与独立性约束，随后应用差异化评估标准：内容谓词采用语义等价，数值谓词要求精确匹配，以模拟实证观察到的人类评估模式。通过引入历史感知的满足函数，DIALEVAL扩展至多轮对话场景，解决了单轮方法失效的问题。验证显示其准确率达90.38%（相比基线降低26.45%错误率），并在复杂指令上与人类判断相关性显著增强。
原文摘要 · Abstract (English)
Evaluating instruction following in Large Language Models requires decomposing instructions into verifiable requirements and assessing satisfaction--tasks currently dependent on manual annotation and uniform criteria that do not align with human judgment patterns. We present DIALEVAL, a type-theoretic framework using dual LLM agents to automate instruction decomposition into typed predicates and implement type-specific satisfaction semantics. The framework enforces formal atomicity and independence constraints during automated extraction, then applies differentiated evaluation criteria--semantic equivalence for content predicates, exact precision for numerical predicates--mirroring empirically observed human assessment patterns. Extended to multi-turn dialogues through history-aware satisfaction functions, DIALEVAL enables evaluation in conversational contexts where single-turn methods fail. Validation demonstrates 90.38% accuracy (26.45% error reduction over baselines) and substantially stronger correlation with human judgment for complex instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。