针对语音编程中转录错误影响代码理解问题,提出多语言语音接口优化方案。
Lost in Transcription: How Speech-to-Text Errors Derail Code Understanding
- 用大模型对语音转录结果进行代码感知修正,提升准确性
- 在4种印度语言和英语上测试,转录错误显著降低任务性能
- 适合需要多语言语音编程的开发者和无障碍工具设计者
代码理解是软件工程工具和开发流程的核心能力。然而,现有系统主要面向使用键盘的英语用户,限制了在多语言及语音优先场景下的可用性,尤其在印度等地区。语音接口虽更具包容性,但涉及代码的语音查询面临非标准英语、领域专有词汇及自定义标识符(如变量名、函数名)等问题,常伴有代码混用表达。本文构建了一个多语言语音驱动的代码理解框架:支持用户以母语说出查询,经自动语音识别(ASR)转录后,利用大语言模型(LLMs)进行代码感知的输出优化,再通过代码模型完成代码问答与检索任务,基于CodeSearchNet、CoRNStack和CodeQA等基准进行评估。聚焦四种广泛使用的印地语系语言与英语,系统分析了转录错误对下游任务的影响,并识别出代码语音识别的关键失败模式。实验表明,大模型引导的修正显著提升了转录与代码理解阶段的表现。研究强调需在语音界面中引入代码敏感适配,并为构建稳健的多语言语音编程工具提供实用解决方案。
原文摘要 · Abstract (English)
Code understanding is a foundational capability in software engineering tools and developer workflows. However, most existing systems are designed for English-speaking users interacting via keyboards, which limits accessibility in multilingual and voice-first settings, particularly in regions like India. Voice-based interfaces offer a more inclusive modality, but spoken queries involving code present unique challenges due to the presence of non-standard English usage, domain-specific vocabulary, and custom identifiers such as variable and function names, often combined with code-mixed expressions. In this work, we develop a multilingual speech-driven framework for code understanding that accepts spoken queries in a user native language, transcribes them using Automatic Speech Recognition (ASR), applies code-aware ASR output refinement using Large Language Models (LLMs), and interfaces with code models to perform tasks such as code question answering and code retrieval through benchmarks such as CodeSearchNet, CoRNStack, and CodeQA. Focusing on four widely spoken Indic languages and English, we systematically characterize how transcription errors impact downstream task performance. We also identified key failure modes in ASR for code and demonstrated that LLM-guided refinement significantly improves performance across both transcription and code understanding stages. Our findings underscore the need for code-sensitive adaptations in speech interfaces and offer a practical solution for building robust, multilingual voice-driven programming tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。