测试大模型函数调用时对格式指令的遵守情况，发现顶级模型仍常出错。
Instruction-Following Evaluation in Function Calling for Large Language Models
- 在JSON Schema中嵌入可验证格式要求，如引号、日期格式等
- 750个测试用例显示，即使GPT-5等顶尖模型也频繁违反格式规范
- 适合评估AI代理系统实际可用性，尤其关注格式一致性
函数调用是大语言模型的核心能力，对AI代理至关重要。现有基准如伯克利函数调用排行榜（BFCL）、tau^2-Bench（arXiv:2506.07982）和ACEBench（arXiv:2501.12851）仅评估参数正确性，未测试参数描述中嵌入的格式指令（如值需加双引号或使用ISO日期格式）。我们提出IFEval-FC，受IFEval（arXiv:2311.07911）启发，评估函数调用中的精确指令遵循能力。IFEval-FC将可验证格式直接编码于JSON Schema描述中，例如指定值不能含标点符号。包含750个测试用例，每个包含一个带有嵌入格式要求的函数及对应用户查询。评估完全算法化，确保客观性、可复现性和可扩展性。结果表明，即使最先进专有模型（如GPT-5、Claude 4.1 Opus）也常未能遵循基本格式规则，凸显其在真实代理系统中的实用性限制。完整代码与数据已公开于https://github.com/Skripkon/IFEval-FC。
原文摘要 · Abstract (English)
Function calling is a core capability of large language models, essential for AI agents. Existing benchmarks such as the Berkeley Function Calling Leaderboard (BFCL), tau^2-Bench (arXiv:2506.07982), and ACEBench (arXiv:2501.12851) evaluate argument correctness but do not test adherence to format instructions embedded in parameter descriptions, such as enclosing values in double quotes or using ISO date formats. We introduce IFEval-FC, a benchmark inspired by IFEval (arXiv:2311.07911) that assesses precise instruction following in function calling. IFEval-FC encodes verifiable formats directly within JSON schema descriptions, for example specifying that a value must not contain punctuation. It includes 750 test cases, each consisting of a function with an embedded format for one of its input parameters and a corresponding user query. Evaluation is fully algorithmic, ensuring objectivity, reproducibility, and scalability. Our results show that even state-of-the-art proprietary models, including GPT-5 and Claude 4.1 Opus, frequently fail to follow basic formatting rules, highlighting a practical limitation for real-world agent systems. The complete codebase and data are publicly available at https://github.com/Skripkon/IFEval-FC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。