How Neural Vocoders Work

Text-to-speech (TTS) technology has come a long way since its inception, making it possible for machines to sound more human-like than ever before. This remarkable progress can be attributed in large part to the advancements in neural vocoders. In this article, we will delve into the basic concepts of neural vocoders, exploring how they work and their role in TTS systems.

The role of neural vocoders in TTS systems

A TTS system is responsible for converting written text into audible speech. It consists of two main components: a text analysis module and a speech synthesis module. The text analysis module converts the input text into a sequence of linguistic symbols, while the speech synthesis module generates the corresponding speech waveforms. The speec synthesis module itself consists of two parts: A synthesizer (such as Tacotron 2), which produces an intermediate spectrogram representation, and a neural vocoder. These neural models play a crucial role: they are responsible for converting intermediate speech representations, such as spectrograms or mel-spectrograms, into the final waveform that we hear as speech.

Tacotron 2 Architecture
The Tacotron 2 Architecture. The Vocoder is shown in green in the top-right (WaveNet MoL)

What is a vocoder?

The term "vocoder" is derived from "voice encoder." Historically, vocoders were used for encoding speech for secure communication or for data compression in telecommunication systems. In the context of TTS, a vocoder is a system that synthesizes speech waveforms from intermediate representations.

From traditional vocoders to neural vocoders

Traditional vocoders, such as the source-filter model and concatenative synthesis, relied on hand-crafted features and signal processing techniques to generate speech. However, these methods often produced robotic-sounding speech due to their limitations in modeling the complex nature of human speech. Neural vocoders, on the other hand, leverage the power of deep learning to model the intricate relationships between the intermediate speech representations and the final speech waveforms. These neural network-based models have been instrumental in achieving more natural-sounding synthesized speech.

Types of neural vocoders

There are several types of neural vocoders, each with its own strengths and weaknesses. Some popular neural vocoder architectures include:


However, despite the significant advancements in TTS technology, there are still challenges to overcome. Neural vocoders can sometimes produce artifacts or unintelligible speech due to the limitations of training data or model architectures. Additionally, real-time synthesis remains a challenge for these models. And finally, speaker similarity (which is what audio deepfakes rely on) is hard, especially when the training data is few or of bad quality.