一个模型搞定六种语言的实时语法高亮,省资源还更准。
Multi Language Models for On-the-Fly Syntax Highlighting
- 用统一模型替代多个单语言模型,减少部署复杂度。
- 通过新归一化技术提升泛化能力,在未见语言上表现更好。
- 少量标注样本即可替代大量生成数据,降低训练成本。
语法高亮是现代开发环境中的关键功能,能显著提升代码可读性和开发效率。然而,由于后端服务存在严格的时间与内存限制,在线和基于Web的开发工具在实时提供准确高亮时仍面临挑战,尤其在代码不完整或无效的情况下。为此,现有系统采用“即席”语法高亮机制,在内容发送前即时生成视觉标记,需应对高请求频率和不完整输入。当前先进方法利用深度学习模仿暴力解析器的行为,通过深度抽象将慢速但准确的解析策略编码为快速统计模型,实现高精度与低延迟。但此类模型存在局限:每模型仅支持一种语言,依赖大规模由慢速解析器生成的数据集,且训练开销大。在多语言环境中,需维护多个独立模型,增加系统复杂性与运维成本。本文提出一种统一模型,可同时支持最多六种主流编程语言,使部署复杂度降低六倍,并在未见语言上表现更优。引入新颖的归一化技术显著提升模型泛化能力,少样本学习实验表明,少量真实样本即可替代大规模生成数据,大幅减少对暴力解析器的依赖。这些创新共同实现了跨多种语言的高效、可扩展、低成本语法高亮。
原文摘要 · Abstract (English)
Syntax highlighting is a critical feature in modern software development environments, enhancing code readability and developer productivity. However, delivering accurate highlighting in real time remains challenging for online and web-based development tools due to strict time and memory constraints on backend services. These systems must serve highlights rapidly and frequently, even when code is partially valid or invalid. This has led to on-the-fly syntax highlighting, where visual annotations are generated just before content is served, often at high request rates and under incomplete input conditions. To meet these demands efficiently, state-of-the-art models use deep learning to learn the behavior of brute-force syntax highlighting resolvers, tools that are easy to implement but too slow for production. Through the Deep Abstraction process, brute-force strategies are encoded into fast statistical models that achieve both high accuracy and low-latency inference. Despite their success, such models face key challenges: they support only one programming language per model, require large datasets from slow brute-force generators, and involve resource-intensive training. In multi-language environments, this means maintaining multiple independent models, increasing system complexity and operational cost. This work addresses these issues by introducing a unified model capable of highlighting up to six mainstream programming languages, reducing deployment complexity by a factor of six and improving performance on unseen languages. A novel normalization technique significantly enhances model generalization, while few-shot learning experiments show that a small number of oracle samples can replace large datasets, minimizing dependence on brute-force generators. Combined, these innovations enable efficient, scalable, and cost-effective syntax highlighting across diverse programming languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。