用自动反馈循环修复视觉语音识别流水线故障,提升稳定性与准确性。
LoopVSR: A Loop Engineering Framework for Automated Repair of Visual Speech Recognition Inference Pipelines

- 通过闭环反馈机制,让代码代理基于真实推理结果自动诊断并修复问题。
- 在CMLR系统中11个主要故障全部修复,平均恢复率达100%。
- 适合需要高可靠性的语音识别系统维护人员或自动化研发团队使用。
视觉语音识别(VSR)在音频嘈杂或缺失时,可通过唇部动作恢复语音。其多阶段推理流水线包括视频解码、嘴部区域提取、预处理、模型调用和解码,上游失败会掩盖下游错误,导致维护仍依赖预设检查和人工调试。本文提出LoopVSR,一种环路工程框架,使代码代理能利用端到端执行证据自动诊断并修复VSR推理流水线。该框架结合受限仓库级诊断与补丁生成,由外部控制器审计变更、运行真实推理,并根据失败情况与字符错误率(CER)决定接受或回滚补丁。反馈循环将新发现的异常、张量统计信息与识别错误返回代理,逐步暴露被上游故障掩盖的问题。在CMLR VSR系统上,LoopVSR以100%平均恢复率修复全部11个主故障,而静态防护仅修复2个,平均恢复率18.13%。此外,它在七次迭代内解决三个级联任务,并在独立的200视频隐藏集上保持恢复能力。结果表明,LoopVSR实现了可度量的端到端自动化修复。
原文摘要 · Abstract (English)
Visual speech recognition (VSR) recovers speech from lip movements when audio is noisy or unavailable. Its multi-stage inference pipeline spans video decoding, mouth-region extraction, preprocessing, model invocation, and decoding, where upstream failures can mask downstream faults. Pipeline maintenance therefore still relies largely on predefined checks and manual debugging. We propose LoopVSR, a Loop Engineering framework that enables a code agent to automatically diagnose and repair VSR inference pipelines using end-to-end execution evidence. It couples constrained repository-level diagnosis and patching with an external controller that audits changes, runs real inference, and accepts or rolls back patches using failures and character error rate (CER). The resulting feedback loop returns newly observed exceptions, tensor statistics, and recognition errors to the agent, progressively exposing faults masked by upstream failures. On the CMLR VSR system, LoopVSR repairs all 11 main faults with 100% mean recovery, whereas the Static guard repairs 2 of 11 with 18.13% mean recovery. It also resolves three cascading tasks in seven accepted iterations and preserves recovery on an independent 200-video hidden set. These results demonstrate that LoopVSR enables measurable, end-to-end automated repair of VSR inference pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。