CLEAR无需掩码即可端到端去除视频字幕，效果更强且支持多语言。
CLEAR: Context-Aware Learning with End-to-End Mask-Free Inference for Adaptive Video Subtitle Removal
- 通过双阶段设计分离字幕表征与生成优化，实现无掩码训练与推理。
- 在中文数据集上提升6.77dB PSNR、降低74.7% VFID，零样本跨6种语言表现优异。
- 仅需基模型0.77%参数量，适合资源受限场景，适合多语言字幕移除任务。
视频字幕去除旨在区分文本叠加与背景内容的同时保持时间连贯性。现有基于扩散的方法在训练和推理阶段均需显式掩码序列，限制了实际部署。本文提出CLEAR（上下文感知的端到端自适应视频字幕去除）框架，实现真正无掩码的端到端推理。其两阶段设计将先验提取与生成精修解耦：第一阶段通过双编码器的自监督正交约束学习解耦字幕表征；第二阶段采用LoRA适配结合生成反馈进行动态上下文调整。方法仅需基扩散模型0.77%参数量进行训练。在中文字幕基准上，相较依赖掩码的基线提升6.77dB PSNR、降低74.7% VFID；并在六种语言（英语、韩语、法语、日语、俄语、德语）上展现优越零样本泛化能力，得益于生成驱动的反馈机制，使推理过程无需真实掩码。
原文摘要 · Abstract (English)
Video subtitle removal aims to distinguish text overlays from background content while preserving temporal coherence. Existing diffusion-based methods necessitate explicit mask sequences during both training and inference phases, which restricts their practical deployment. In this paper, we present CLEAR (Context-aware Learning for End-to-end Adaptive Video Subtitle Removal), a mask-free framework that achieves truly end-to-end inference through context-aware adaptive learning. Our two-stage design decouples prior extraction from generative refinement: Stage I learns disentangled subtitle representations via self-supervised orthogonality constraints on dual encoders, while Stage II employs LoRA-based adaptation with generation feedback for dynamic context adjustment. Notably, our method only requires 0.77% of the parameters of the base diffusion model for training. On Chinese subtitle benchmarks, CLEAR outperforms mask-dependent baselines by + 6.77dB PSNR and -74.7% VFID, while demonstrating superior zero-shot generalization across six languages (English, Korean, French, Japanese, Russian, German), a performance enabled by our generation-driven feedback mechanism that ensures robust subtitle removal without ground-truth masks during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。