用聊天机器人设计音频编码器替代图像生成文本编码器,测试其有效性。
Testing chatbots on the creation of encoders for audio conditioned image generation
- 让五个公开聊天机器人设计音频编码器,替代Stable Diffusion的CLIP文本编码器。
- 所有设计均在超200万音频-图像-文本对上训练,但无一达到理想效果。
- Gemini编码器量化表现最佳,Grok生成图像更连贯,适合后续专项研究。
近年来,聊天机器人在编程任务中日益流行。然而,现代生成式图像模型仍主要依赖文本编码器将语义概念转换为视觉表征,尽管音频也可作为输入。本文探索了先进对话模型是否能设计出有效音频编码器,以替代Stable Diffusion 1.5中的CLIP文本编码器,实现直接从声音生成图像。我们向五个公开可用的聊天机器人提出任务,要求设计神经网络架构作为音频编码器,并设定明确共享条件。每个有效提议的编码器均在超过两百万个相关音频-图像-文本观测数据上训练,并在保留验证集和测试集上通过多种指标评估,辅以生成图像的定性分析。尽管几乎所有聊天机器人均生成了有效的模型结构,但无一取得令人满意的结果,表明其音频嵌入未能可靠对齐原始文本编码器。其中,Gemini提出的音频编码器在量化指标上表现最佳,而Grok编码器生成的图像更具连贯性(尤其与原文本编码器配合时)。研究揭示了聊天机器人间的共通架构偏差,凸显未来模型需弥补的编码能力鸿沟。我们还公开了演示系统,供公众研究和测试这些音频编码器。最后,我们提出未来应开展更聚焦、高度专业化任务,以充分检验聊天机器人的创造力与推理能力。
原文摘要 · Abstract (English)
On one hand, recent advances in chatbots has led to a rising popularity in using these models for coding tasks. On the other hand, modern generative image models primarily rely on text encoders to translate semantic concepts into visual representations, even when there is clear evidence that audio can be employed as input as well. Given the previous, in this work, we explore whether state-of-the-art conversational agents can design effective audio encoders to replace the CLIP text encoder from Stable Diffusion 1.5, enabling image synthesis directly from sound. We prompted five publicly available chatbots to propose neural architectures to work as these audio encoders, with a set of well-explained shared conditions. Each valid suggested encoder was trained on over two million context related audio-image-text observations, and evaluated on held-out validation and test sets using various metrics, together with a qualitative analysis of their generated images. Although almost all chatbots generated valid model designs, none achieved satisfactory results, indicating that their audio embeddings failed to align reliably with those of the original text encoder. Among the proposals, the Gemini audio encoder showed the best quantitative metrics, while the Grok audio encoder produced more coherent images (particularly, when paired with the text encoder). Our findings reveal a shared architectural bias across chatbots and underscore the remaining coding gap that needs to be bridged in future versions of these models. We also created a public demo so everyone could study and try out these audio encoders. Finally, we propose research questions that should be tackled in the future, and encourage other researchers to perform more focused and highly specialized tasks like this one, so the respective chatbots cannot make use of well-known solutions and their creativity/reasoning is fully tested.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。