How Text-to-Speech Synthesis Works, From WaveNet to Voice Clones

How text-to-speech synthesis works

Modern text-to-speech (TTS) turns written text into audio in three steps. A text front end normalises the input, expanding "£25" into "twenty-five pounds" and resolving abbreviations, and may convert words into phonemes. An acoustic model then predicts how the sentence should sound, frame by frame, usually as a mel spectrogram that encodes pitch, timing and timbre. Finally, a vocoder turns that spectrogram into a waveform you can play.

Each step is now a neural network trained on recorded speech paired with transcripts. The acoustic model learns where to place stress and pauses from examples rather than from hand-written rules, and the vocoder learns how real speech fills in the detail a spectrogram leaves out. Add a speaker embedding taken from a short clip of someone's voice, and the same pipeline becomes a voice cloning system.

That last point is why TTS belongs in any discussion of deepfakes. The technology that reads articles aloud and powers accessible interfaces is the same technology used to make a scam call sound like a manager or a relative.

What TTS sounded like before neural networks

Until the mid-2010s, commercial systems used one of two methods. Concatenative TTS stitched together small units cut from hours of recordings by a single voice actor. It could sound natural within a sentence but produced audible joins, and changing the voice meant recording a new actor. Parametric TTS generated speech from a statistical model of acoustic features and passed it through a signal-processing vocoder. It was flexible and compact but sounded muffled or buzzy.

Neither method made convincing impersonation practical. Cloning a person meant recording them for hours in a studio, which ruled out using a few clips from social media.

Why WaveNet changed synthetic speech in 2016

WaveNet changed synthetic speech by generating the waveform directly with a neural network, one sample at a time, instead of reconstructing it from hand-designed acoustic features. DeepMind researchers submitted the paper on 12 September 2016. Each audio sample was predicted from all the samples before it, which let the model learn the fine structure of real speech, including breathing and lip sounds that earlier vocoders discarded.

In the paper's listening tests for US English and Mandarin Chinese, people rated WaveNet speech as more natural than the best parametric and concatenative systems. The same model could produce many voices by conditioning on speaker identity, an early hint of how cloning would later work.

The original was far too slow for products. DeepMind's October 2017 production version was about 1,000 times faster and generated one second of 24 kHz speech in 50 milliseconds, which put neural speech into Google Assistant and made it a mainstream expectation.

How Tacotron changed the text-to-speech pipeline

Tacotron changed TTS by replacing the hand-built text analysis and acoustic modelling stages with one network trained end to end. Google researchers submitted "Tacotron: Towards End-to-End Speech Synthesis" on 29 March 2017. It read characters and produced spectrogram frames directly, learning pronunciation, timing and intonation from text and audio pairs alone, and scored a mean opinion score of 3.82 out of 5 on US English, beating a production parametric system for naturalness.

Tacotron 2, submitted on 16 December 2017, joined that idea to a WaveNet vocoder. A sequence-to-sequence network predicted mel spectrograms from characters, and a modified WaveNet turned them into audio. It scored 4.53 against 4.58 for professionally recorded speech. That two-stage design, spectrogram predictor plus neural vocoder, became the default for the next several years and the base of early zero-shot cloning systems.

From Tacotron to three-second voice clones

Two later steps turned high-quality TTS into a fraud tool. In 2018 Google's SV2TTS work conditioned a Tacotron 2 synthesiser on an embedding from a speaker-verification encoder, so a few seconds of reference speech could steer the voice without retraining. In 2020 GAN vocoders such as HiFi-GAN made generation much faster than real time on a single GPU, so synthetic speech could keep pace with a conversation.

In January 2023 Microsoft's VALL-E treated speech as discrete codec tokens and predicted them with a language model trained on 60,000 hours of English, cloning an unseen voice from a 3-second enrolled recording. The reference audio needed to impersonate someone had shrunk from studio hours to a voicemail greeting.

How synthetic TTS speech is detected

Each stage of the pipeline leaves traces. Acoustic models produce unusually regular prosody and timing; vocoders and codecs leave fingerprints in phase and high-frequency structure. Detectors learn these patterns from large sets of real and synthetic speech, and the ASVspoof challenges, most recently ASVspoof 5 in 2024, measure how well they generalise to attacks they have not seen.

Neural Defend's NeuroCall, currently in Beta, applies this to call audio and returns a synthetic-speech risk signal for your verification policy. NeuroVerify accepts audio in the same API request as face images and video during onboarding or account recovery. Neither replaces process controls: a TTS clone that passes detection should still fail a callback to a number on file.