Best Deepfake Detector Evaluation Criteria: An Open Test Standard

What are the best deepfake detector evaluation criteria?

A detector is the best choice only for a stated job, and the criteria below describe that job in measurable terms. Each one has a metric you can compute yourself and a piece of evidence you can ask a vendor to hand over. If a criterion has no metric, it is a preference, and it belongs in the commercial part of the decision, not the technical part.

Weight the criteria by the cost of each mistake in your workflow. A bank onboarding flow that sends every flagged selfie to manual review can tolerate more false positives than a platform that blocks uploads automatically. The same detector can win one comparison and lose the other.

Why accuracy and AUC alone mislead buyers

NIST AI 100-4, published in November 2024, notes that accuracy, AUROC and F1 are the figures most often reported for synthetic content detectors, and that some experts argue they miss what matters most: in many settings a false positive, where real content is called synthetic, causes serious harm. NIST points to true positive rate at a capped false positive rate as an alternative, and stresses that whether an AUROC of 0.95 is good enough depends on the decision the score feeds.

VendorBench-100 shows the gap in practice. The benchmark ran 36 systems on one adversarial set of 100 still images and ranked them by MCC because, as the authors state, accuracy and F1 are unreliable on that imbalanced corpus. TruthScan reached a ROC-AUC of 0.923, close to the top, yet its MCC was 0.322 and its accuracy 0.49 at its default threshold. Hive showed the same pattern with a ROC-AUC of 0.90 and an MCC of 0.290. A ranking score tells you the detector can separate classes somewhere along its curve. MCC tells you whether it does so at the threshold you will actually run.

How can I check whether a vendor's deepfake accuracy claims are real?

Ask for the conditions behind the number, then reproduce it on media the vendor has not seen. A claim such as 99 percent accuracy is uninterpretable until you know the dataset, the ratio of real to fake samples, the threshold, the model version and the test date. A detector that labels everything as fake scores 90 percent accuracy on a set that is 90 percent fake.

Then test for distribution shift. Deepfake-Eval-2024, an in-the-wild benchmark first posted to arXiv in March 2025, found that open-source detectors lost about 50 percent of their AUC on video, 48 percent on audio and 45 percent on images compared with older academic benchmarks. A score earned on FaceForensics-style data says little about the selfies, call recordings or claim photos your users submit.

Finally, rerun any public result yourself where the corpus is available. That is the strongest test of a claim, and it costs a few hours.

• Which dataset, which version, and what share of samples were synthetic?
• What threshold produced the figure, and is it the threshold the API ships with?
• What is the full confusion matrix, including specificity on real media?
• Which model version was tested, on what date, and has it changed since?
• Were unsupported files, timeouts and errors excluded from the denominator?

How to run a fair head-to-head comparison

Write the protocol before any vendor sees the test media. Fix the sampling plan, label rules, metrics, thresholds and pass marks, and keep the held-out set in your custody. NIST AI RMF 1.0 asks for documented, repeatable test and evaluation processes, and a frozen protocol is the simplest way to meet that bar.

Give every candidate the same files, the same time limit and the same interface rules. Pin model and SDK versions. Count unsupported files, timeouts and errors as outcomes in the results, because production traffic will contain all three. If one vendor gets extra configuration help or a second attempt, record it and offer the same to the others.

Report results by media type, attack class and transformation, with sample counts. With zero errors on n samples, the 95 percent upper bound on the true error rate is roughly 3 divided by n, so a clean run on 100 fakes still allows a miss rate near 3 percent. Size each slice for the precision you need.

Where content-based detection helps and where it stops

A deepfake detector answers one question: does this media carry signs of generation or manipulation? It does not establish who submitted it, whether a camera feed was injected before capture, or whether a real photo is being used out of context. NIST AI 100-4 concludes that none of the content transparency techniques it reviews, including detection, watermarking and provenance metadata, is a comprehensive solution on its own.

Evaluate each capability against the question it is built to answer. Provenance checks such as C2PA validation, presentation attack detection under ISO/IEC 30107-3 and injection defences need their own tests. Detection output is a risk signal, and your policy decides whether a flagged file is rejected, reviewed or allowed.

How Neural Defend performs against these criteria

Neural Defend ranked first of 36 systems on the public VendorBench-100 benchmark (v2, July 2026), with an MCC of 0.876 and a ROC-AUC of 0.979. The next-best vendor, Reality Defender, scored an MCC of 0.646. Neural Defend's precision was 0.963, recall 0.987 and specificity 0.857. One VendorBench author, Sharayu N. Deshmukh, also co-authored Neural Defend's ARCAS 1B paper, so we do not describe the benchmark as independent. The corpus and results are public, and anyone can rerun them.

VendorBench-100 covers still images only. It does not measure video, audio, injection attacks or production latency, so treat it as one input to your own test. The ARCAS paper reports pooled accuracy for ARCAS 1B, the model behind Neural Defend's products, of 99.99 percent on GenImage, 99.92 percent on WildFake and 97.65 percent on OpenFake.

In production, NeuroVerify returns six signals in one API call, including face manipulation detection and AI generative model analysis, with a unified risk score and signal-level evidence your policy can act on. Eligible image requests complete in under 200 ms at p95 under defined test conditions. Bring your own held-out media to the evaluation and score us with the protocol above.