arXiv:2608.15510cs.AI2026-08

让视觉和代码分支各司其职,动态分配贡献度,提升图表转代码准确率。

Who Leads Now? Token-Level Modality Arbitration for Chart-to-Code Generation

论文配图:Who Leads Now? Token-Level Modality Arbitration for Chart-to-Code Generation
图 1 · 摘自论文原文
  • 采用双路径结构,视觉与代码分支独立处理,通过轻量仲裁器动态分配权重。
  • 在三个基准上表现优于通用与专用模型,推理过程与最终代码均获奖励优化。
  • 适合需要高精度图表转代码的开发者或自动化工具研究者使用。

图表转代码需模型细读图表视觉细节并生成可执行代码。现有方法或分开训练视觉与编码能力,或在图表-代码数据上联合微调,使两种能力纠缠。这既未考虑两者本质差异,也导致优化冲突。本文提出MoCA(跨模态仲裁混合模型),将两种能力分离而非融合。MoCA基于跨模态仲裁块(CAB),包含独立的视觉分支与代码分支,以及一个轻量级仲裁器,在每一层和每个生成标记处动态决定两者的贡献。训练分两阶段:先在自蒸馏推理轨迹上进行监督预热,将视觉理解分解为显式步骤;再通过强化学习,对推理过程与最终代码同时给予奖励。分析表明,仲裁器学习到的是有结构而非随机的分配策略,专家贡献随标记、层级和实例系统性变化。在三个基准上,MoCA性能媲美通用与图表专用模型。消融实验显示,性能提升并非仅因模型更大,而是来自互补的分支初始化与输入条件下的仲裁机制协同作用。

原文摘要 · Abstract (English)

Chart-to-code generation requires a model to read the fine-grained visual details of a chart and write executable code that reproduces it. Existing chart-to-code methods either train visual and coding abilities separately, or fine-tune on chart-to-code data with the two abilities entangled. Neither strategy accounts for the distinct nature of the two abilities or the interference that arises when they are optimized together. We propose MoCA (Mixture of Cross-modal Arbitration), which separates the two abilities rather than blending them. MoCA is built on Cross-modal Arbitration Block (CAB), which maintains a visual branch and a code branch as two distinct pathways, and a lightweight arbiter that arbitrates their relative contributions at every layer and generated token. We train MoCA in two stages: a supervised warm-up on self-distilled reasoning trajectories that decomposes visual understanding into explicit steps, followed by reinforcement learning with rewards on both the reasoning process and the final code. Analysis shows that the arbiter learns structured rather than arbitrary allocations, with expert contributions varying systematically across tokens, layers, and instances. Across three benchmarks, MoCA delivers competitive performance against general-domain and chart-specialized models. Ablation results show that the gains cannot be attributed to a larger model size alone, but instead arise from the joint contributions of complementary visual and code branch initialization and input-conditioned arbitration through CAB.

图表转代码跨模态动态分配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。