How AI Voice Conversion Works and How It Differs From Text-to-Speech

How voice conversion works

AI voice conversion takes a recording or live stream of one person speaking and re-renders it in another person's voice, keeping the words, timing, intonation and emotion of the original speaker. The source speaker drives the performance; the system swaps the vocal identity.

Most converters follow the same three-part logic. First, a content encoder strips speaker identity from the source audio and keeps what was said and how: phonetic content, rhythm and pitch contour. Second, a representation of the target voice is taken from reference recordings. Third, a decoder and vocoder combine the two and generate a waveform that carries the source's delivery in the target's timbre.

The hard part is the first step. If the content representation still carries traces of the source speaker, the output sounds like a blend of two people. Much of the research since 2019 has been about separating content from identity cleanly, without needing parallel recordings of both speakers saying the same sentences.

Three approaches that shaped modern voice conversion

AUTOVC, published in May 2019 and presented at ICML 2019, showed that a plain autoencoder with a carefully sized information bottleneck could separate content from speaker identity using only a reconstruction loss. Its authors presented it as the first zero-shot voice conversion system, able to convert to a voice it had not been trained on and to work on non-parallel data.

Self-supervised speech models made separation easier. Models pre-trained on large amounts of unlabelled speech produce frame-level features that encode phonetic content well. kNN-VC, published in May 2023 and accepted at Interspeech 2023, used these features in a strikingly simple way: it replaces each frame of the source with its nearest matching frame from the target speaker's reference audio, then runs a pretrained vocoder. Its authors report that it works with just a few examples of the target speaker.

Real-time converters used by hobbyists and attackers combine these ideas with fast GAN vocoders, so the converted voice can keep pace with a conversation on ordinary hardware.

What is the difference between text-to-speech and voice conversion deepfakes?

Text-to-speech deepfakes start from typed text and the machine decides how to say it; voice conversion deepfakes start from a human speaking and the machine only changes whose voice it sounds like. That difference shapes both the attack and the defence.

Where voice conversion is used legitimately

Voice conversion has a long history outside fraud. Film and game studios use it to keep a character's voice consistent when a different actor records pickup lines, and dubbing teams use it to carry an actor's own voice into another language. Speech technology researchers use it to help people whose voices have changed through illness speak in a voice closer to their own, and to anonymise speakers in research recordings.

These uses share two conditions: the person whose voice is produced has agreed, and listeners are told when it matters. The EU AI Act's Article 50, which applies from 2 August 2026, turns the second condition into law for deepfake audio, requiring deployers to disclose it to the people who hear it.

Why voice conversion is the harder attack to spot

Many listener tips for spotting clones target text-to-speech weaknesses: flat emotion, odd pauses, trouble with unexpected questions. Voice conversion removes those weaknesses because a real person is speaking. The attacker can laugh, interrupt, hesitate and improvise, and the converter supplies the target's timbre.

Law enforcement warnings reflect this. The FBI's May 2025 alert on AI voice messages impersonating senior US officials said clones can sound nearly identical to the real person and that listeners should look for call lag or mismatched voice patterns rather than rely on tone. Conversion adds a small processing delay, which can show up as lag, but network delay on ordinary calls makes this an unreliable tell.

The practical conclusion is the same as for any voice deepfake. A voice on its own should never authorise a payment, password reset or account change. Callback to a number on file, pre-agreed code words and out-of-band confirmation work whether the voice was typed or performed.

How voice conversion deepfakes are detected

Detection focuses on what the converter adds rather than on prosody. Content encoders, nearest-neighbour matching and vocoders each leave statistical traces in spectral detail and phase. Anti-spoofing benchmarks include voice conversion attacks alongside text-to-speech; the 2024 ASVspoof 5 challenge used crowdsourced recordings from many speakers and added adversarial attacks designed to evade detectors.

Neural Defend's NeuroCall, currently in Beta, screens call audio for synthetic speech, including converted voices, and returns a risk signal for your call-handling policy. NeuroDetect supports investigators reviewing recordings, with audio-segment evidence and a forensic report, and NeuroVerify accepts audio alongside face images and video in one API request during onboarding and account recovery. Like any detector, they are most reliable on clean audio and should be tested on your own call conditions.

When you evaluate any detector against voice conversion, build a test set that reflects the real attack: converted speech driven by different human speakers, played over the telephony codecs and networks your customers use, with background noise and short utterances. Include genuine calls from many accents and age groups so you can measure false positives fairly. Score results with metrics that hold up on imbalanced data, and agree in advance what happens to a flagged call.