arXiv:2607.18704cs.SDcs.AI2026-07

将语音转为可搜索结构化内容,全程透明可追溯。

What the Waveform Knows: Transparent-first Speech and Audio Intelligence with Caption Studio

论文配图:What the Waveform Knows: Transparent-first Speech and Audio Intelligence with Caption Studio
图 1 · 摘自论文原文
  • 基于Whisper和pyannote构建语音转写与说话人分离核心
  • 直接从音频信号提取波形、语调、停顿等30+特征
  • 每项指标标明来源,适合需要可解释性的企业应用

Caption Studio 是一个以透明性为核心的语音与音频智能平台,通过自动转录、说话人分离、语音分析、信号级音频分析及字幕生成,将语音和视频转化为结构化、可搜索的内容。系统采用FastAPI后端与实时仪表盘,包含三层架构:(i) 基于Whisper类自动语音识别和pyannote说话人分离的转写与分离核心;(ii) 音频智能层,直接从音频信号中提取波形、频谱图、音高、语速、静默段、填充词频率、情感等声学与语言特征;(iii) 集成层,支持数据导出与下游流程集成。本文的核心贡献是透明性优先框架,所有报告指标均明确标注为测量值、推导值或不可用,显著提升语音分析的可追溯性、可解释性与可靠性。论文还介绍了系统架构、基准测试方法、可解释性与不确定性框架,以及企业级部署的关键考量。

原文摘要 · Abstract (English)

Caption Studio is a transparency-first speech and audio intelligence platform that transforms spoken audio and video into structured, searchable content through automated transcription, speaker diarization, speech analytics, signal-level audio analysis, and subtitle generation. The system is built on a FastAPI backend with a real-time dashboard and adopts a three-layer architecture comprising (i) a transcription and diarization core based on Whisper-class automatic speech recognition and pyannote speaker diarization, (ii) an audio intelligence layer that extracts acoustic and linguistic features, including waveforms, spectrograms, pitch, speaking rate, silence, filler-word frequency, and sentiment, directly from the audio signal, and (iii) an integration layer that supports data export and downstream workflow integration. A principal contribution of this work is the transparency-first framework, in which every reported metric is explicitly identified as measured, derived, or unavailable, thereby improving the traceability, interpretability, and reliability of speech analytics. The paper presents the system architecture, benchmarking methodology, explainability and uncertainty framework, and key considerations for enterprise-scale deployment.

语音分析透明性音频智能可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。