CookVoice: An Unified Framework for Multi-Modal Style Controllable Human Voice Generation
Speech generation/ Singing Voice Generation/ Multimodal-Style Control/ Multimodal-Prosody Control.
1UNSW Sydney 2Dolby Laboratories
Abstract
Human voice generation has made rapid progress in speech generation, singing voice generation, voice cloning, and voice editing. However, most existing systems are designed for specific tasks and often rely on task-dependent architectures, control signals, or autoregressive decoding, limiting fine-grained controllability and inference efficiency. In this paper, we propose CookVoice, a unified framework for multimodal, multi-style, and multi-task human voice generation. CookVoice decomposes the human voice into three key factors: content, prosody, and style, enabling both speech and singing voice generation within a unified model. To achieve precise and flexible controllability, we design a flexible alignment strategy that maps text, style, and prosody control signals onto the frame-level of spectrogram. This design allows CookVoice to support a wide range of tasks, including text-to-speech, text-to-singing voice, style-controllable generation, voice mimicry, voice conversion, and voice editing. Experimental results show that CookVoice achieves generation quality comparable to existing Text-to-Speech and text-to-singing voice baselines, while providing stronger style and prosody controllability. Moreover, CookVoice achieves comparable performance to large-scale baselines with only 43.51 million parameters and efficient inference using as few as 4 ODE steps, making it a practical solution for real-world human voice generation applications.
Contents
1. CookVoice Speech Demos
The following samples show bilingual TTS, text and voice style control,
and word-level speech editing. Replace the audio files under
assets/audio/ with generated samples.
1.1 Bilingual Speech Generation
| Prompt | Text | Audio |
|---|---|---|
| A female in adult age is speaking English with neutral emotion | I am CookVoice, a unified generative model for human voice generation | |
| A female in adult age is speaking Chinese with neutral emotion | 我是一个统一的人声生成模型 |
1.2 Style Controllable TTS
CookVoive support voice style control through natural language description and voice-based zero-shot control.
1.2.1 Text Style
| Style Prompt | Text | Prompt Value | EN | ZH |
|---|---|---|---|---|
| A [AGE] female is speaking [LANGUAGE] | I didn't expected to see you here/没想到能在这里见到你 | Child | ||
| Teenager | ||||
| Young Adult | ||||
| Adult | ||||
| Senior | ||||
| A female is speaking [LANGUAGE] in [EMOTION] emotion | I haven't seen you in the classroom, what are you doing here?/我在教室里面没看到你,你在这里做什么？ | Sad | ||
| Neutral | ||||
| Happy | ||||
| Angry | ||||
| A [GENDER] is speaking [LANGUAGE] | Happy Birthday to you! This is the gift for you! /生日快乐!这是给你的生日礼物！ | Female | ||
| Male |
1.2.2 Voice-Style
| Text | Reference Voice | Generated Speech |
|---|---|---|
| 枯藤老树昏鸦，小桥流水人家 | ||
| The stars are like a thousand little promises shining in the sky | ||
1.3 Speech Editing
| Task | Original | Operation | Edited | Diagram |
|---|---|---|---|---|
| Word replacement |
I am going to vist andy with my mother. |
Changing "andy" to "kevin" and "mother" to "father" |
I am going to vist kevin with my father. |
→
|
| 你好，我们一起坐火车去北京玩吧 | Changing "火车" to "飞机" and "北京" to "上海" | 你好，我们一起坐飞机去上海玩吧 |
→
|
|
| Duration Control |
I am so hungry, I can eat a cow! |
Expand the duration of "so", shorten "hungry", expand "cow" |
I am so hungry, I can eat a cow! |
→
|
| 哇！我看到了一辆好帅的汽车！ | Expand the duration of "哇，好帅", shorten "汽车" | 哇！我看到了一辆好帅的汽车！ |
→
|
2. Singing Voice Generation
2.1 Score to Singing Voice Generation
| Source | Style Control | Prosody Control | Sample | Figure |
|---|---|---|---|---|
| Lyric: 怎么可以，拥有你 | Text: a male with alto voice is singing a chinese song. the song is singing with sad emotion, fast pace, low vocal range, vibrato singing technique | Note |  |
|
| F0 |  |
|||
| Voice | Note |  |
||
| F0 |  |
|||
| Lyric: i wanna catch on fire it's buried in my soul | Text: a male with alto voice is singing a english song. the song is singing with sad emotion, fast pace, medium vocal range, glissando singing technique | Note |  |
|
| F0 |  |
|||
| Voice | Note |  |
||
| F0 |  |
2.2 Singing Voice Editing
| Source | Task | Operation | Sample |
|---|---|---|---|
| Lyric: 想你时你在心田 | Lyric Editing | Change "心田" to "哪里" |
→
|
| Style Editing | Change female voice to male voice |
→
|
|
| Duration Editing | Expand "心" shorten "田" |
→
|
|
| Prosody Editing | Combine the last three note, and increase the Note from 62 to 65 |
→
|
|
| Lyric: falls to the ground may these memories break our fall | Lyric Editing | Change "memories" to "dreams" and "our" to "your" |
→
|
| Style Editing | Change male voice to female voice |
→
|
|
| Duration Editing | Expand "break" shorten "memories" |
→
|
|
| Prosody Editing | Change the last note from 69 to 61 |
→
|
2.3 Prosody Control
| Input Type | Visualization | Audio |
|---|---|---|
| NOTE |
|
|
| Sketch F0-1 |
|
|
| Sketch F0-2 |
|