用神经网络存储视频音频，实现高效压缩与浏览器实时解码
Data-driven Video Codec with Implicit Neural Representations
- 将视频音频转为坐标到像素/音量的神经网络映射
- 2.33MiB压缩后仍保持28.72dB视频质量，16位量化即达峰值
- 支持浏览器内端到端训练与解码，适合在线流媒体场景
传统编码器将视频存储为压缩像素数据。本文则将视频及其音频轨道存储为单一正弦表示网络（SIREN）的权重，该网络将时空坐标映射为RGB值和音频振幅。网络采用独立的音视频初始化层、共享全连接隐藏层堆叠，以及三个输出分支：一个视频分支和两个孪生音频分支，其差异用于估计并去除残差噪声。过拟合的教师网络通过基于响应的知识蒸馏压缩为更小的学生网络，再经16位对称权重量化和无损LZMA2（xz）编码。在6.08 MiB测试视频上，量化后的学生网络实现28.72 dB视频PSNR、0.75 SSIM，音频PSNR达24.18 dB，对数谱距离10.69 dB，整体表示从9.05 MiB压缩至2.33 MiB，压缩比2.61。1~32位量化实验显示，重建质量在16位时饱和。对比H.264、HEVC和MP3，指出其在部分指标上的不足，并展示基于WebRTC的浏览器原型，可实现模型的训练、传输与解码。
原文摘要 · Abstract (English)
A conventional codec stores a video as compressed pixel data. We instead store the video, together with its audio track, as the weights of a single sinusoidal representation network (SIREN) that maps space-time coordinates to RGB values and audio amplitudes. The network uses separate audio and video initialization layers, a stack of shared fully connected hidden layers, and three output branches: one for video and two Siamese audio branches whose disagreement is used to estimate and subtract residual noise. The overfitted teacher network is then compressed by response-based knowledge distillation into a smaller student, followed by 16-bit symmetric weight quantization and lossless LZMA2 (xz) encoding. On a 6.08 MiB test video, the quantized student reaches a video PSNR of 28.72 dB with SSIM of 0.75, and an audio PSNR of 24.18 dB with a log spectral distance of 10.69 dB, while the pipeline shrinks the representation from 9.05 MiB to 2.33 MiB, an overall compression ratio of 2.61. A bit-width sweep from 1-bit to 32-bit quantization shows that reconstruction quality saturates at 16 bits. We compare against H.264, HEVC, and MP3, report where the approach falls short of them, and describe a browser-based prototype that trains, transfers, and decodes these models over WebRTC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。