When Should a Deepfake Detector Send a Case to Human Review?

When should a deepfake detector send a case to human review?

Human review is expensive and slow, so it should be reserved for cases where a person adds information the model lacks. Four triggers cover most workflows. The first is an uncertain score: set a band between the auto-approve and auto-decline thresholds, sized from your own labelled data so that the expected review volume is affordable. The second is disagreement between signals, for example a clean liveness result with a high face-manipulation score. The third is poor input, such as a tiny face, heavy compression or very short audio, where the model's validation data says little. The fourth is consequence: declining an applicant, freezing an account or referring a case to police deserves a person's sign-off whatever the score.

Add a fifth, quieter route: a small random sample of automatic approvals and declines. Without it you only ever review the cases the model already doubted, and you never learn about confident mistakes.

Do heatmaps and explainability make deepfake detection results trustworthy?

They make results easier to inspect, which is different from making them correct. A heatmap shows which regions most influenced the model's output. That helps a reviewer check whether the model focused on the face boundary, the eyes or a background artefact. It does not show that the model's reasoning is sound.

Adebayo and colleagues, in Sanity Checks for Saliency Maps (2018), tested popular saliency methods by randomising model weights and training labels. Some methods produced nearly the same maps afterwards, which means the maps reflected the image more than the model. Visual plausibility alone was misleading. For deepfake review, the lesson is to treat explanations as pointers, confirm them against what the reviewer can see in the original media, and prefer tools that state their uncertainty and keep the model version with every explanation.

What should a human review workflow for deepfake detection include?

Reviewers need a bounded evidence package and a defined set of outcomes. The package should include the original media at full resolution, the capture metadata, per-signal scores and the unified score, any regions, frames or audio segments flagged, input-quality warnings, the model and policy versions, and the business context of the request. Outcomes should be a short list, such as genuine, manipulated, insufficient evidence and escalate, each with a required reason.

Order matters. People anchor on numbers, and the EU AI Act's Article 14 on human oversight names this risk directly as automation bias, the tendency to over-rely on a system's output. A practical counter is to show the media first, record the reviewer's independent impression, and then reveal the model evidence. Disagreements between the first impression and the final call are useful training data.

How good are human reviewers at spotting deepfakes?

Untrained people are poor judges of modern synthetic faces and voices, while trained forensic analysts remain a strong reference. Deepfake-Eval-2024, a benchmark of in-the-wild deepfakes collected in 2024, found that commercial detectors and models fine-tuned on its data outperformed off-the-shelf open-source models but did not yet reach the accuracy of deepfake forensic analysts. Review queues therefore work best when staffed by people trained on current attack types, with the hardest cases escalated to specialists.

Measure reviewers as you would a model. Track agreement between reviewers on a shared sample, overturn rates against later confirmed outcomes, and time to decision. Rotate staff who review explicit or distressing content, such as intimate image abuse, and give them blurring and audio-muting tools by default.

Which controls keep a review queue from becoming a bottleneck or a rubber stamp?

A review queue fails in two ways: cases wait too long, or reviewers approve everything to keep up. Both are preventable with explicit controls.

• Set a maximum queue age per route and escalate cases that exceed it, rather than letting them auto-approve.
• Require a second approver for declines and referrals above a value or impact threshold.
• Cap reviewer workload per hour and watch for decision times that fall far below the team median.
• Feed confirmed outcomes back to threshold tuning so the review band shrinks where the model is reliable.
• Keep a record of each decision, reason, reviewer and time, linked to the evidence record.

How NeuroDetect supports human review

NeuroDetect is Neural Defend's forensic workspace for exactly this step. A reviewer opens a case, sees the original media alongside explainability heatmaps and attention views down to regions, frames and audio segments, and reads an LLM-assisted plain-language summary in which each statement links back to the evidence. The analyst makes the final call. Every analysis keeps its model and version context, and the downloadable forensic report records observations, uncertainty, provenance, model context and analyst notes.

Upstream, NeuroVerify supplies the routing inputs: a unified risk score with signal-level evidence from six signals, so a policy engine can detect signal conflicts and send them to the right queue. The detection behind both products, built on the ARCAS 1B model, ranked first of 36 systems on the public VendorBench-100 still-image benchmark with an MCC of 0.876.