Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretraining

Jinlong Xue^†, Yayue Deng^†, Yingming Gao, Ya Li^*
Beijing University of Posts and Telecommunications, Beijing, China

Abstract

Recent prompt-based text-to-speech (TTS) models can clone an unseen speaker using only a short speech prompt. They leverage a strong in-context ability to mimic the speech prompts, including speaker style, prosody, and emotion. Therefore, the selection of a speech prompt greatly influences the generated speech, akin to the importance of a prompt in large language models (LLMs). However, current prompt-based TTS models choose the speech prompt manually or simply at random. Hence, in this paper, we adapt retrieval augmented generation (RAG) from LLMs to prompt-based TTS. Unlike traditional RAG methods, we additionally consider contextual information during the retrieval process and present a Context-Aware Contrastive Language-Audio Pre-training (CA-CLAP) model to extract context-aware, style-related features. The objective and subjective evaluations demonstrate that our proposed RAG method outperforms baselines, and our CA-CLAP achieve better results than text-only retrieval methods.

Model Architecture
Evaluation

Model Architecture

Figure 1: An overview structure of RAG-TTS

Figure 2: Left: An Overview of Context-Aware CLAP. Right: An illustration of the inference process of Prompt-based TTS

Evaluation

Groundtruth: the groundtruth speech.
Self: use the groundtruth text-audio pair as prompt, it serves as upper bound of prompt-based TTS performance.
Random: randomly select one text-audio pair as prompt text and prompt speech.
MiniLM: adopt text-only embedding model all-MiniLM-L6-v2 for retrieval.
CoROM: adopt text-only embedding model corom_sentence-embedding_chinese-base for retrieval.
Proposed: our proposed method using Context-Enhanced CLAP for retrieval.