研究LLM在不完美视觉验证下如何改进TikZ代码定制效果
Imperfect Visual Verification for Code Edition : A Case Study on TikZ
- 构建基于不完美验证器的迭代编辑框架，评估反馈有效性
- 验证器F1最高达0.815，能较准确判断视觉指令是否生效
- 精准反馈可显著提升弱模型定制成功率，强模型更受益于高质量验证
大型语言模型在代码生成方面取得显著进展，但针对TikZ等生成视觉结果的程序，尤其是视觉代码定制任务仍具挑战。此类任务要求局部化、语义保持的修改：模型需定位相关代码，按指令修改并保留结构与渲染效果。现有方法依赖后验迭代修正，由验证器提供反馈引导修正，但在视觉输出难以形式化评估的场景下，确定性验证器不存在，开发者只能依赖不完美验证器。本文以TikZ为案例，聚焦其弱代码结构、细粒度视觉语义和特征定位难等问题，将视觉代码定制建模为带不完美预言机的迭代编辑问题。提出分析框架，开展大规模实验，评估多种基于LLM及工具增强的视觉验证器在迭代修正流程中的表现，并通过大量人工标注修正轨迹分析验证器行为与反馈质量。结果显示，即使验证器不完美，仍可实现中等精度判断（最高F1=0.815），反馈显著提升迭代修正效果，尤其对Qwen3-vl-30b-a3b-Instruct等弱模型带来11–20次完美定制提升；更强模型如Gemini-3获益较少（+5），但更依赖准确验证避免过早接受错误结果。反馈有效性的关键在于精确识别图像问题、提供可操作指导、覆盖所有相关问题且紧扣原始指令。
原文摘要 · Abstract (English)
LLMs have significantly advanced code generation, enabling the synthesis of functional programs. While recent systems achieve strong performance on many coding benchmarks, tasks involving programs such as TikZ that generate visual artifacts remain challenging, in particular on visual code customization. Unlike generation from scratch, customization requires localized, semantics-preserving edits: the model must locate relevant code, modify it according to the instruction, and preserve the remaining structure and rendering. Approaches based on post-hoc iterative refinement/correction where a verifier provides feedback to guide corrections, have shown promise. However, in the case of programs with a visual outcome such as in TikZ, where correctness is harder or likely impossible to formalize and evaluate automatically, deterministic verifiers do not exist. Hence, developers can only rely on imperfect verifiers. In this paper, we conduct an empirical study to answer:to what extent can iterative refinement remain effective when the verifier itself is unreliable?} We use TikZ as a focused case study that isolates the core difficulties of the problem (weak code structure, fine-grained visual semantics, and difficult feature localization) in a controlled and challenging setting. We define visual code customization as an iterative editing problem with an imperfect oracle, and introduce a framework for analyzing such iterative refinements. We conduct a large-scale study and evaluate multiple LLM-based and tool-augmented visual verifiers within iterative refinement pipelines, and perform extensive manual annotation of refinement trajectories to assess verifier behavior and feedback quality. Our findings show that even imperfect verifiers can determine with moderate accuracy whether visual instructions are applied to code, achieving F1-scores up to 0.815. Feedback improves iterative refinement, especially for weaker models, adding 11--20 perfect customizations for Qwen3-vl-30b-a3b-Instruct, while stronger models like Gemini-3 gain fewer improvements (+5) but benefit more from accurate verification that prevents premature acceptance. Feedback is effective only when it precisely identifies image issues, provides actionable guidance, addresses all relevant problems, and remains grounded in the original instruction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。