用可解释的损失训练卷积同步网络，提升大图处理与音频视频对齐精度。
Interpretable Convolutional SyncNet
- 采用受BCE和InfoNCE启发的平衡二元交叉熵损失，避免复杂采样。
- 在LRS2和LRS3数据集上分别达到96.5%和93.8%的同步准确率，为当前最优。
- 输出具概率解释性，支持定义偏移概率、离屏率等新评估指标，适合质量评估场景。
由于真实视频常因各种原因出现音画不同步，需使用同步网络将其恢复同步以用于需要同步视频的任务。现有最先进（SOTA）的同步网络依赖InfoNCE损失或变压器架构，或两者兼有。然而，InfoNCE损失使模型输出难以解释，而变压器架构对大图像不友好，限制了同步网络的实际应用。本文提出使用平衡二元交叉熵损失（BBCE）训练卷积型同步网络，该损失结合了二元交叉熵（BCE）和InfoNCE的优点，无需复杂的采样策略。所提模型能更好处理大尺寸图像，且输出具备概率意义。这一特性使得我们可定义偏移概率和离屏比率等度量来评估音视频语音数据集的同步质量。实验表明，该模型在LRS2数据集上达到96.5%的准确率，在LRS3数据集上达到93.8%的准确率，优于现有方法。
原文摘要 · Abstract (English)
Because videos in the wild can be out of sync for various reasons, a sync-net is used to bring the video back into sync for tasks that require synchronized videos. Previous state-of-the-art (SOTA) sync-nets use InfoNCE loss, rely on the transformer architecture, or both. Unfortunately, the former makes the model's output difficult to interpret, and the latter is unfriendly with large images, thus limiting the usefulness of sync-nets. In this work, we train a convolutional sync-net using the balanced BCE loss (BBCE), a loss inspired by the binary cross entropy (BCE) and the InfoNCE losses. In contrast to the InfoNCE loss, the BBCE loss does not require complicated sampling schemes. Our model can better handle larger images, and its output can be given a probabilistic interpretation. The probabilistic interpretation allows us to define metrics such as probability at offset and offscreen ratio to evaluate the sync quality of audio-visual (AV) speech datasets. Furthermore, our model achieves SOTA accuracy of $96.5\%$ on the LRS2 dataset and $93.8\%$ on the LRS3 dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。