用一句话控制人物形象和声音,实现音画同步的个性化生成。
ID-LoRA: Identity-Driven Audio-Video Personalization with In-Context LoRA
- 通过上下文LoRA联合建模音视频,支持文本+图片+音频共同控制
- 语音相似度提升24%,在不同环境条件下表现更稳定
- 仅需3000对数据训练,适合资源有限的研究者使用
现有视频个性化方法虽能保留视觉特征,但音视频分离处理。音频模型缺乏视觉场景信息,无法与画面动作同步;传统语音克隆仅依赖参考录音,无法通过文本改变语调或声学环境。我们提出ID-LoRA(身份驱动的上下文LoRA),在单一模型中联合生成主体外观与语音,由文本提示、参考图像和短音频片段共同控制双模态输出。该方法基于LTX-2音视频扩散模型,采用参数高效的上下文LoRA进行适配,据我们所知是首个实现音视频一次性个性化生成的方法。两个关键挑战:参考与生成标记共享位置编码空间,难以区分;说话人特征在去噪过程中易被稀释。为此,我们引入负时间位置编码,将参考标记置于独立RoPE区域,同时保持其内部时序结构;提出身份引导机制,通过对比有无参考信号的预测结果,增强说话人特异性特征。人类偏好评估显示,73%的标注者更偏好ID-LoRA的语音相似度,65%更认可其语调风格。跨环境设置下,说话人相似度较Kling提升24%,且差异越大,优势越明显。初步用户研究表明,联合生成为物理合理的声音合成提供了有益归纳偏置。整个方法仅需约3000对训练数据,在单个GPU上完成训练。代码、模型与数据将公开发布。
原文摘要 · Abstract (English)
Existing video personalization methods preserve visual likeness but treat video and audio separately. Without access to the visual scene, audio models cannot synchronize sounds with on-screen actions; and because classical voice-cloning models condition only on a reference recording, a text prompt cannot redirect speaking style or acoustic environment. We propose ID-LoRA (Identity-Driven In-Context LoRA), which jointly generates a subject's appearance and voice in a single model, letting a text prompt, a reference image, and a short audio clip govern both modalities together. ID-LoRA adapts the LTX-2 joint audio-video diffusion backbone via parameter-efficient In-Context LoRA and, to our knowledge, is the first method to personalize visual appearance and voice in a single generative pass. Two challenges arise. Reference and generation tokens share the same positional-encoding space, making them hard to distinguish; we address this with negative temporal positions, placing reference tokens in a disjoint RoPE region while preserving their internal temporal structure. Speaker characteristics also tend to be diluted during denoising; we introduce identity guidance, a classifier-free guidance variant that amplifies speaker-specific features by contrasting predictions with and without the reference signal. In human preference studies, ID-LoRA is preferred over Kling 2.6 Pro by 73% of annotators for voice similarity and 65% for speaking style. On cross-environment settings, speaker similarity improves by 24% over Kling, with the gap widening as conditions diverge. A preliminary user study further suggests that joint generation provides a useful inductive bias for physically grounded sound synthesis. ID-LoRA achieves these results with only ~3K training pairs on a single GPU. Code, models, and data will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。