用图像表示文字，让语音合成更抗生僻字和跨语言挑战
Pixel-TTS: Image based Text Rendering for Robust Text-to-Speech
- 把文本转成图像，通过卷积层提取视觉特征生成语音嵌入
- 在零样本场景下表现稳定，收敛速度比传统方法快
- 适合需要跨语言、抗字形变异的语音合成应用
基于像素的文本建模研究表明，将文本表示为图像可使模型利用视觉线索理解语言。将文本与其视觉形式对齐，使结构相似但编码不同的字符产生相近的嵌入，有利于跨语言与零样本场景。传统基于文本的方法独立处理每个字符，限制了对未见字符的泛化能力，且跨语言适配需扩展嵌入矩阵。我们提出 Pixel-TTS，一种基于视觉表征的语音合成框架。该框架将文本渲染为图像，并通过二维卷积层投影生成嵌入。此设计避免了微调时的嵌入矩阵扩展，同时提升对未见字符和拼写变异的鲁棒性。大量实验表明，Pixel-TTS 在性能上媲美强基线，收敛更快，且具备出色的零样本泛化能力。
原文摘要 · Abstract (English)
Recent advances in pixel-based text modeling show that representing text as images enables models to exploit visual cues for language understanding. Grounding text in its visual form allows structurally similar characters with different Unicode encodings to produce similar embeddings, benefiting cross-lingual and zero-shot scenarios. Conventional text-based approaches treat each character independently, limiting generalization to unseen characters and requiring embedding expansion during cross-lingual adaptation. We propose Pixel-TTS, a text-to-speech framework for visually grounded speech synthesis. It renders text as images and projects them through a 2D convolutional layer to generate embeddings. This design eliminates embedding matrix expansion during fine-tuning while improving robustness to unseen characters and orthographic variations. Extensive experiments show Pixel-TTS achieves competitive performance with strong baselines, faster convergence and robust zero-shot generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。