Get Large Language Models Ready to Speak: A Late-fusion Approach for Speech Generation

TTS-Llama and MoLE-Llama are advanced text-to-speech systems. TTS-Llama, using a fine-tuned Llama model, excels in speech synthesis. MoLE-Llama, a multimodal language model, integrates text and speech through late-fusion, addressing catastrophic forgetting and outperforming TTS-Llama on text QA tasks. Both models demonstrate potential in text-speech multimodal applications, showcasing improvements in speech generation and text normalization.