Audio Deepfake Detection Architecture: How Detectors Work

Audio deepfake detection architecture at a glance

An audio deepfake detection system is a pipeline with five stages: ingest and normalise the audio, find the speech worth analysing, extract features, score each segment with a trained classifier, then aggregate and calibrate those scores into a result a policy can use. Each stage exists because real audio is messy. Calls arrive in different codecs and sample rates, recordings contain silence and music, and synthetic speech may cover only part of a file.

The table below shows what each stage does and the failure it prevents. The rest of this page explains the design choices inside each one.

How do audio deepfake detectors work?

Audio deepfake detectors work by learning, from large labelled sets of real and synthetic speech, which acoustic patterns separate the two, then scoring new audio against those patterns. The patterns are mostly below what people hear: phase behaviour, fine harmonic structure, high-frequency energy and the regularities that particular vocoders and codecs introduce.

Two families of front end dominate. The first feeds the classifier a time-frequency representation or the raw waveform and lets it learn artefacts directly. AASIST, submitted in October 2021, is the reference example: it uses graph attention across spectral and temporal domains to model artefacts in both at once, and its authors reported a 20% relative improvement over the previous best system, with a light variant of only 85,000 parameters.

The second family starts from a self-supervised speech model pre-trained on large amounts of unlabelled speech. A February 2022 paper fine-tuned wav2vec 2.0 as the front end, combined it with data augmentation and reported the lowest equal error rates then published on the ASVspoof 2021 logical access and deepfake tasks. Self-supervised front ends tend to generalise better to unseen generators, at a higher compute cost.

Training choices matter as much as architecture. Contrastive objectives, which pull bona fide samples together in embedding space and push spoofed ones away, and heavy augmentation with codecs, noise and reverberation are common ways to make detectors hold up when the audio differs from the training set.

Segment-level scoring and aggregation

Score short windows, not whole files. A partial fake may replace three words inside a ten-minute genuine call, and a file-level average will bury it. Typical systems score overlapping windows of a few seconds, keep the per-window results, and report both an overall score and the time ranges that drove it.

Aggregation then has to be explicit. Taking the maximum window score catches partial fakes but raises false positives on noisy calls; averaging is calmer but misses short insertions. Many deployments use a rule that requires several high-scoring windows, or a learned aggregator, and calibrate the final score so a value of 0.8 means roughly the same thing across channels and over time.

Every result should carry the model version, the segments analysed and any regions excluded, such as music or crosstalk. Without that record, a reviewer cannot tell a confident synthetic finding from a score produced on two seconds of muffled speech.

Live call analysis versus recorded audio

Live analysis streams audio in small chunks and must return a usable signal within the conversation. It favours lighter models, rolling windows and early provisional scores that firm up as more speech arrives. The FTC's 2024 Voice Cloning Challenge recognised a winning approach that scored incoming call audio in two-second chunks, which shows the shape of a real-time design.

Recorded analysis can afford heavier models, multiple passes and full-file context. It suits fraud review, disputes and investigations, where an analyst needs evidence more than speed.

Telephony needs special care in both cases. Narrowband codecs cut frequencies above about 3.4 kHz, where some synthesis artefacts sit, and packet loss concealment adds artefacts of its own. Train and test on audio passed through the codecs you actually receive, otherwise a model that looks excellent on studio clips will disappoint on calls.

Measuring and governing an audio detector

Research reports equal error rate (EER) because it summarises a detector in one number, but production teams need more: false positive rate at the operating threshold, performance per channel and per language, and behaviour on generators released after training. ASVspoof 5, published in August 2024, made this harder on purpose by adding crowdsourced recordings from many more speakers and, for the first time, adversarial attacks built to fool detectors.

Treat the detector as a monitored component. Log score distributions by channel, sample flagged and unflagged calls for review, and retest when a new voice generator becomes popular. NIST's AI 100-4 report on synthetic content (November 2024) frames detection as one of several transparency tools alongside provenance and watermarking, and that framing is a good guide: combine signals, and keep a person or a policy in charge of the decision.

How Neural Defend applies this architecture

Neural Defend's NeuroCall, currently in Beta, applies segment-level synthetic speech detection to call audio and returns a risk signal for call-handling rules. NeuroVerify accepts audio in the same API request as face images and video, returning one unified risk score with signal-level evidence for onboarding and recovery. NeuroDetect gives analysts audio-segment views and a downloadable forensic report that records model and version context.

The detection models build on ARCAS 1B, Neural Defend's own deepfake detection model. For teams that cannot send recordings outside their network, deployment options include private cloud or VPC, secure container, on-premises and air-gapped installs, and Neural Defend holds ISO/IEC 27001, ISO/IEC 27701, ISO/IEC 42001 and SOC 2 Type II certification.