How AI Face Swaps Work: What a Face Swap Deepfake Is and How It Is Detected

What is a face swap deepfake?

A face swap deepfake is an image or video in which one person's face has been replaced with another person's face by a machine learning model. The original performer's head movement, expression, body, clothing, background and often voice remain; only the apparent identity changes.

That distinguishes face swaps from other face manipulations. FaceForensics++, a 2019 benchmark by Rössler and colleagues, grouped manipulation methods into identity swaps (DeepFakes and FaceSwap) and facial reenactment (Face2Face and NeuralTextures). Reenactment keeps the person and changes what their face does. Lip-sync changes only the mouth to match new audio. Full face synthesis invents a face that belongs to nobody. The distinction matters for detection, because each method leaves different traces.

How AI face swaps work, step by step

Most face swap pipelines follow the same five stages, whether they run offline on recorded video or live on a webcam stream.

The classic approach, popularised by open tools such as DeepFaceLab (Perov et al., first posted in 2020), trains a model on many images of both people. A shared encoder learns to describe any face as a compact code covering pose, expression and lighting, and a separate decoder for each person learns to rebuild that person's face from the code. Feeding person A's code into person B's decoder produces B's face making A's expression.

Newer one-shot methods remove the need for training on the target. SimSwap (Chen et al., 2021) extracts an identity representation from a single source image and injects it into the target's features with an ID Injection Module, while a weak feature matching loss helps keep the target's expression and gaze. That is what makes fast, low-effort swaps possible, including real-time swaps streamed through a virtual camera.

• Detect and align: find the face in each frame and locate landmarks such as eyes, nose and jawline so faces can be normalised.
• Encode: describe the face's pose, expression and lighting, and separately capture the identity to be inserted.
• Generate: render the new identity with the original pose, expression and lighting.
• Blend: place the generated face back using a mask, matching colour and smoothing the seam.
• Smooth over time: in video, reduce flicker between frames so the swap holds as the head moves.

What traces does a face swap leave?

Each stage of the pipeline can leave evidence. The blend stage is the most common source: a boundary around the inner face where texture, sharpness or colour shift slightly, sometimes visible at the jawline, hairline or ears. The generate stage can produce skin that is smoother or more uniform than the rest of the image, or lighting on the face that does not match the scene.

Identity mismatches are another clue. A swap changes the inner face but usually keeps the original head shape, ears, hair and neck, so the face can look subtly wrong for the head it sits on. In video, temporal inconsistency appears as flicker, momentary slips during fast movement, and failures when a hand or object passes in front of the face or the head turns to profile.

Some traces are invisible to people. Generators leave statistical patterns in pixel frequencies and noise that trained models can pick up. FaceForensics++ reported that detectors trained on manipulated images outperformed human observers and stayed accurate under strong compression, although compression, resizing and re-recording all erode the signal.

Where face swaps show up in fraud

Identity verification is the largest target. FinCEN's alert FIN-2024-Alert004 of 13 November 2024 describes criminals altering or generating images to get through banks' customer identification, and notes that some use third-party webcam plugins to display previously generated video instead of a live feed. A swapped face on a stolen identity can then pass a face match against the victim's genuine ID.

Remote hiring is another. The FBI's June 2022 notice on deepfakes in remote job applications reported interviews where the applicant's lip movements did not fully match the audio, and where coughs and sneezes did not line up with what was on camera. Executive impersonation is the third: the FBI's December 2024 announcement describes real-time video chats impersonating company executives and police. The guide to face swap detection for KYC covers the identity case in operational detail.

How face swaps are detected

Reliable face swap detection combines several checks, because no single trace survives every swap method and every compression setting. Models examine boundary consistency, texture statistics, frequency patterns, landmark geometry and, in video, motion and temporal coherence. Liveness checks add a separate question, whether a live person is present, and can be passed by a swap streamed in real time.

Standards now expect manipulation analysis in identity flows. NIST SP 800-63A-4, published in July 2025, requires remote identity proofing to analyse submitted media for manipulation and recommends analysing it for signatures of generative AI and deepfake tools.

Human review still has a role, mainly for context: does the scene, voice and story fit? It should not be the main defence against high-quality swaps, which may show no flaw a viewer can see.

How Neural Defend detects face swaps

For investigators and analysts, NeuroDetect shows where a swap was found. An authorised image or video is uploaded into a case, and explainability heatmaps and attention views point to specific regions and frames, with a downloadable report recording observations, uncertainty and model version. That makes it suited to verifying a suspect clip of an executive or a disputed recording.

For automated identity checks, NeuroVerify runs Face Manipulation Detection alongside five other signals, including Live Presence Verification and AI Generative Model Analysis, in one API call, returning eligible image results in under 200 ms p95 under defined test conditions.

Both build on ARCAS 1B, Neural Defend's own detection model. The public VendorBench-100 benchmark includes face swaps among its eight edge-case families; across its 100 still images, Neural Defend ranked first of 36 systems with an MCC of 0.876 against 0.646 for the next vendor. One VendorBench author also co-authored the ARCAS paper, and the benchmark does not test video, where temporal traces add further evidence.