arXiv:2602.10735cs.SDcs.AI2026-02

开源工具Calliope让电子书自动配音并精准同步,支持离线使用。

Calliope: A TTS-based Narrated E-book Creator Ensuring Exact Synchronization, Privacy, and Layout Fidelity

  • TTS生成时直接捕获音频时间戳,确保语音与文字精确对齐。
  • 完整保留原文排版、样式和嵌入媒体,不丢失任何格式细节。
  • 全程离线运行,保护隐私且无云端费用,适合教育与无障碍阅读。

有声电子书将同步音频与数字文本结合,播放时高亮当前朗读的词句,有助于早期识字和阅读障碍者,也方便普通读者在阅读与听书间切换。随着自然音质神经网络文本转语音(TTS)技术的发展,已有多个商业服务可将标准文本电子书转换为高质量有声书。但目前尚无开源解决方案。本文提出Calliope,一个开源框架,利用最先进的开源TTS技术将文本电子书转换为符合EPUB 3 Media Overlay格式的有声电子书。该方法创新性地在TTS生成过程中直接记录音频时间戳,确保语音与文字显示完全同步;严格保留原始排版、样式和嵌入媒体;整个流程可在本地离线执行。此设计避免了持续的API费用、隐私风险及云服务的版权合规问题。当前支持XTTS-v2和Chatterbox两款主流开源TTS系统。相较先生成音频再通过强制对齐同步的方案,我们的方法能实现精确同步,实验表明强制对齐会产生显著的音文偏移,影响阅读体验。源代码与使用说明见https://github.com/hugohammer/TTS-Narrated-Ebook-Creator.git。

原文摘要 · Abstract (English)

A narrated e-book combines synchronized audio with digital text, highlighting the currently spoken word or sentence during playback. This format supports early literacy and assists individuals with reading challenges, while also allowing general readers to seamlessly switch between reading and listening. With the emergence of natural-sounding neural Text-to-Speech (TTS) technology, several commercial services have been developed to leverage these technology for converting standard text e-books into high-quality narrated e-books. However, no open-source solutions currently exist to perform this task. In this paper, we present Calliope, an open-source framework designed to fill this gap. Our method leverages state-of-the-art open-source TTS to convert a text e-book into a narrated e-book in the EPUB 3 Media Overlay format. The method offers several innovative steps: audio timestamps are captured directly during TTS, ensuring exact synchronization between narration and text highlighting; the publisher's original typography, styling, and embedded media are strictly preserved; and the entire pipeline operates offline. This offline capability eliminates recurring API costs, mitigates privacy concerns, and avoids copyright compliance issues associated with cloud-based services. The framework currently supports the state-of-the-art open-source TTS systems XTTS-v2 and Chatterbox. A potential alternative approach involves first generating narration via TTS and subsequently synchronizing it with the text using forced alignment. However, while our method ensures exact synchronization, our experiments show that forced alignment introduces drift between the audio and text highlighting significant enough to degrade the reading experience. Source code and usage instructions are available at https://github.com/hugohammer/TTS-Narrated-Ebook-Creator.git.

TTS有声书开源离线处理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。