About this website

This website presents a variety of aspects regarding audio synthesis, deepfakes, and deepfake detection. Maybe most importantly, we show results of current state-of-the-art audio synthesis algorithms:

We create a set of sentences which are spoken by an artificial intelligence (AI). However, the AI does not simply reproduce audio recordings based on the training data; it creates entirely new speakers. Thus, for every sentence you hear, the corresponding speaker (along with its vocal tonality) does not exist!

To achieve this, we are using deep learning. More precisely, we are using a deep-learning text-to-speech (TTS) model. These models work by learning how speech relates to spoken audio via a dataset <text, audio>. Such datasets are presented to the learning model, which after about 2-5 days of training can speak new sentences.

If you listen closely, you can make out artefacts in the synthesized speech. For example, you can notice that the tonality or prosody of the sentences does not always match the semantics, i.e. the content. This is because despite all advances in AI, machine-learning is at its core 'pattern recognition', not true intelligence. The TTS model does learn the relationship between text and audio, but it does not 'understand' the meaning of the sentence.

Resources

• The Tacotron model, the godfather of neural TTS. This model is also called 'synthesizer', and creates a mel-spectogram from given text input.
• The Wavenet model, another core contribution in neural TTS. This model is also called 'vocoder', and converts the synthesizer's mel-spectogram to time-audio representation which humans can listen to.
• Creating new speakers can be achieved via a multi-speaker architecture, such as presented here. They key idea is to train a model using a multi-speaker dataset and embedding speaker characteristics into latent vector representation. During inference, one can then create new speakers by sampling a new, random speaker vector.
• Coqui AI, formerly Mozilla TTS, a popular Github project featuring many TTS models.
• A cool random sentence generator I used for this project.
• VITS , the TTS model I used to synthesize the sentences. It is an end-to-end model, thus combining synthesizer and vocoder.

Please feel free to contact me if you have any questions.