用可配置的结构相似度评分器,精准衡量大模型输出的JSON与标准格式的匹配程度。
Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization
- 通过递归对齐树结构,结合匈牙利算法与序列比对,实现结构化评分
- 在多个数据集上提升提示优化效果,且不引入额外开销
- 支持图结构数据的引用对齐,可自动修复错误并生成建议
大型语言模型常被要求生成符合固定JSON Schema的输出,用于信息抽取、工具调用、智能体规划和知识图构建。衡量输出与标准参考之间的匹配程度至关重要,但难度极大:精确匹配过于脆弱,文本相似性忽略结构,而使用LLM作为评判者成本高、不透明且非确定性。我们提出Object Aligner(OA),一个开源Python库,通过递归对齐两个JSON对象的树结构(无序集合采用匈牙利算法,有序集合采用序列对齐),并在模式声明的粒度上给予部分得分。该方法完全通过JSON Schema扩展进行配置,适应新任务只需标注模式而非编写代码。然而,复杂结构数据很少是扁平树:记录可能形成以任意标识符为键的图或超图,破坏了先前相似度度量的假设。我们的核心贡献——引用对齐,通过推断黄金与候选标识符间的双射关系,使评分对重命名保持不变。由于精确恢复双射属于图同构问题,OA通过Weisfeiler-Leman颜色精炼近似求解。有序敏感的序列模式适用于排序与规划任务。由于同一对齐可定位所有不匹配项,OA无需额外成本即可生成排名修复建议。将其作为奖励函数集成至GEPA提示优化器中,可在所有数据集上有效提升或保持中立表现。
原文摘要 · Abstract (English)
Large language models (LLMs) are often asked to produce JSON conforming to a fixed schema, powering information extraction, tool calling, agentic planning, and knowledge-graph construction. Measuring how closely an output matches a gold reference is essential yet surprisingly hard: exact match is brittle, text similarity ignores structure, and an LLM judge is expensive, opaque, and non-deterministic. We address this with Object Aligner (OA), an open-source Python library that scores two JSON objects deterministically by recursively aligning their trees (the Hungarian algorithm for unordered collections, sequence alignment for ordered ones) and awarding partial credit at the granularity the schema declares. The Object Aligner is configured entirely through a set of JSON Schema extensions, so adapting it to a new task involves annotating a schema rather than writing code. Complex structured data, however, are rarely flat trees: records may form graphs or hypergraphs keyed by arbitrary identifiers, breaking the assumptions of prior similarity metrics. Our central contribution, referential alignment, closes this gap by inferring a bijection between gold and candidate identifiers and scoring every reference through it, so the score is invariant to relabeling. Since recovering this bijection exactly is graph isomorphism, the Object Aligner approximates it with Weisfeiler-Leman color refinement. An order-sensitive sequence regime targets ranking and planning. Since the same alignment localizes every mismatch, the Object Aligner emits ranked repair suggestions at no extra cost. Used as a reward inside the GEPA prompt optimizer, Object Aligner helps or stays neutral across all datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。