Abstract
Zero-Shot Text-To-Speech(TTS) has become a more challenging speech generation task, but current TTS systems lack flexible generation over emotional information. This paper proposes a customized emotion Zero-Shot TTS system based on multi-modal prompt. The system extracts the emotion information from the arbitrary modal prompt(text, image or speech) and models the content, timbre, emotion and prosody separately. Firstly, we propose a multi-modal prompt emotion encoder based on the pretrained Emotion2Vec to extract the emotion information from the prompts. Then, we introduce a prosody predictor based on autoregressive(AR) transformer to fit the distribution of prosody and propose the emotion consistency loss(ECL) to retain emotion information in the predicted prosody. Finally, we introduce the diffusion-based acoustic model to generate target mel-spectrogram. The subjective and objective experiments show that our system outperforms similar systems in naturalness and similarity, and achieves more flexible emotion and timbre customization.
Emotion Controlling Based On Speech Prompt
( Note that: in the 'Ours w/o MPEE', we replaced the proposed MPEE with the AMPE of MM-TTS.)
| don't ask me to carry an oily rag like that. | |||||||
| she had your dark suit in greasy wash water all year. | |||||||
| he ate four extra eggs for breakfast. | |||||||
| it was not whatever tale was told by tails. |
Emotion Controlling Based On Image Prompt
 |
no price is too high when true love is at stake. | ||||
 |
don't ask me to carry an oily rag like that. | ||||
 |
this was easy for us. | ||||
 |
destroy every file related to my audits. |
Emotion Controlling Based On Text Prompt
| A man talks in a tone of happiness. | are your grades higher or lower than nancy's. | ||||
| A man voices with a pitch of intense gloom. | she had your dark suit in greasy wash water all year. | ||||
| A woman speaks with a tone of extreme anger. | no price is too high when true love is at stake. | ||||
| A man voices with a slight hint of annoyance. | why do we need bigger and better bombs? |