评测模型能否精准修复SVG代码中的视觉缺陷而不影响其他部分。
Vector-Bench: Can Models Surgically Edit SVG Code?

- 构建40个高难度SVG修复任务,要求模型按指令修改且保持其余元素不变。
- 最强模型仅15.0%完全符合规范,平均修复进度达43.7%但语义保真度低。
- 适合关注视觉生成与代码编辑一致性的研究者使用。
基于指令的矢量编辑需具备两个能力:完成指定修改,同时确保其余内容完全不变。后者在仅以位图评估输出时极易被忽略。我们提出Vector-Bench,一个包含40个SVG修复任务的小型但高难度基准。每个任务包含一个损坏的SVG程序、作者编写的一条视觉指令、隐藏的目标程序、平均每任务5.05个标注修复,以及平均60.55个需保护的对象。指令描述可见缺陷,不暴露元素标识符、坐标、颜色码或路径数据。定义确定性二值规范奖励:请求修改采用属性感知的感知容差,未请求的渲染或应用相关结构必须语义不变,结果必须为合法SVG。保留规范目标等价性和更严格的源保真度作为诊断工具。有效性门控的修复进展、近完整的层级结构,以及有效输出的意外变更率(UCR)用于解释部分结果。我们在1360次请求上评估了34个模型端点(25个开源权重,5个低成本控制组,4个前沿闭源端点)。最强模型仅达到15.0%的完整规范成功,尽管平均修复进度为43.7%,表明表面修复进度与规范忠实编辑仍有显著差距。所有提示、输出、评分代码、成本及每任务报告均已公开。
原文摘要 · Abstract (English)
Instruction-based vector editing requires two capabilities: making a requested change and leaving everything else alone. The second is easy to miss when an output is judged only as a raster image. We introduce Vector-Bench, a compact, difficult benchmark of 40 SVG repair tasks. Each task pairs a corrupted SVG program with an author-written visual instruction, a hidden target program, 5.05 annotated repairs on average, and an average of 60.55 protected objects. Instructions describe visible defects without exposing element identifiers, coordinates, color codes, or path data. We define a deterministic binary specification reward: requested repairs use attribute-aware perceptual tolerances, while unrequested rendering- or application-relevant structure must remain semantically unchanged and the result must be a valid SVG. Canonical target equality and stricter source fidelity are retained as diagnostics. Validity-gated repair progress, a near-complete tier, and valid-output Unintended Change Rate (UCR) explain partial outcomes. We evaluate 34 model endpoints (25 listed as open-weight, 5 inexpensive controls, and 4 frontier closed endpoints) over 1360 requests. The strongest endpoint reaches only 15.0% full specification success, despite 43.7% mean repair progress, showing that apparent repair progress and specification-faithful editing remain substantially different. All prompts, outputs, scoring code, costs, and per-task reports are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。