What is a forensic tool's error rate?

What the factor actually asks

Daubert’s third factor is a single sentence containing two questions: the known or potential rate of error, and the existence and maintenance of standards controlling the technique’s operation.

Almost every exchange about tool error rates in a digital-evidence case is conducted as though the factor stopped after the first clause. It does not, and the omission is expensive for whichever side made it. Counsel demanding a percentage is asking for the one thing the discipline cannot supply; an examiner who answers only that percentage question has left the half with a documentary answer on the table.

There is a second reason the factor is misapplied here. It was written for a technique applied repeatedly to comparable specimens, where a false-positive rate is a measurable property of the technique. A forensic examination is a sequence of decisions about which records to read on one machine. The nearest thing to a testable technique inside it is a single tool function — image this device, parse this structure, recover from this filesystem — and that is exactly the level at which forensic tool testing is actually done.

Why a forensic tool usually has no published error rate

Three structural reasons, none of them evasions.

A suite is not a technique

A commercial forensic platform images drives, mounts filesystems, carves files from unallocated space, parses registry hives, decodes SQLite databases, indexes text, and reconstructs browser history. Those functions fail in unrelated ways. Imaging either produces a bit-for-bit copy that verifies against its source hash or it does not; carving reconstructs content from fragments and is wrong in degrees. A single figure spanning both would mean nothing.

The input distribution is not defined

An error rate needs a population to be a rate over. For a laboratory assay the population is specimens of a known type. For a forensic parser it would have to be “all the filesystems, all the application versions, all the states of partial corruption a real device can be in” — a population nobody can enumerate, let alone sample. Testing against a constructed reference set answers a narrower and more honest question: did this version behave as specified on material whose correct answer was known in advance.

That is what a reference dataset is for, and it is why the credible claim is conformance to a specification rather than an accuracy percentage.

Most of what is disputed is not the tool

When an opposing report is genuinely wrong, the tool usually displayed exactly what the structure contained. The error sits one step later, where a record of one thing was described as evidence of another. A published tool error rate would not have caught it, which is the strongest practical argument that the factor is not where a reliability fight belongs.

Three different things litigators call an error rate

Untangling these is most of the work, because an objection aimed at one of them is answered with material about another and everybody leaves the exchange believing they won it.

An objection aimed at one of them is answered with material about another, and everybody leaves the exchange believing they won it.

The practical test on receiving an error-rate objection is to ask which row it belongs in. If it is row one, produce the testing record. If it is row two, produce the second tool’s output. If it is row three — which it usually is, whatever words were used — then the tool is not the subject and the argument should be conducted where it actually lives.

What have courts accepted instead?

The reported rulings are more forgiving than the briefing usually assumes, and they are consistent about why: Daubert’s factors are non-exclusive, and a court may find reliability from other indicia.

• A low potential rate. In Williford v. State, the reliability showing for an imaging product included that it is generally accepted in the field, is commercially available and therefore testable by anyone, has been tested, has been the subject of published comparisons, and has a low potential rate of error. That five-part proffer is still the pattern, and it can be assembled for any current product.
• No error rate as to one narrow operation. In United States v. Chiaradio, the district court acknowledged the tracing program had not been independently tested, and relied instead on the agent’s specialised experience and his ability to re-create the program’s output by hand. The absence of peer review carried little weight because the inquiry is flexible, and the source code was deliberately secret.
• A conceded figure for a hash comparison. In United States v. Collins, the defence’s own examiner agreed with the government agent that SHA-1 values are in excess of 99.9999 percent accurate, and the challenge was abandoned. Expect a hash attack to be traded away rather than won, and know the figure you would concede.
• A human verification step. In State v. Roberts, what carried the reliability showing for a hash-matching toolkit was that the method was explained and that officers independently reviewed the identified files rather than acting on the tool’s output directly.
• Training and use rather than internals. In State v. Pratt the foundation was 800-plus hours of forensic training, a week of dedicated training on that tool, and use of it on hundreds of devices. In Krause v. State the examiner could not explain how the hash algorithm worked and was admitted anyway, because he could explain what a matching hash establishes and that he had verified one.

What does validation actually look like in an engagement?

Validation is the word that does the work the error rate cannot. It has three tiers, and a report should be explicit about which tier each load-bearing finding sits at.

1. 01Tool-level. The product and version used for each step, and the public testing record for that version where one exists. This is the tier NIST CFTT reports occupy, and it is written once and reused, because it is a property of the software rather than of the matter.
2. 02Function-level, in house. Running the tool against material whose correct answer is already known — a reference image, a device the examiner populated deliberately — and recording that the output matched. This is what turns “the tool is reliable” into “this installation, on this workstation, produced the right answer on known data.”
3. 03Finding-level. For each conclusion the opinion rests on, a second route to the same result: another independently developed parser, a manual read of the structure, or a different artifact that records the same event. This is the tier that answers the analytical-gap objection, and it is the only one that cannot be prepared in advance.

Finding-level validation is also what the reported rulings reward. The Seventh Circuit in United States v. Owens found the opinion connected to the data by more than the expert’s say-so because four independent records pointed the same way — matching info hashes, a hash match for every one of the 226 pieces of the file, the installed client version, and a most-recently-used folder entry. No single one of those carried the conclusion.

What to write down while it is happening

• The tool, the version, and the parser or signature definition date at each step — versions change parsing behaviour between releases, and a finding from an older build may not reproduce on a newer one.
• The acquisition log as produced by the imaging tool, including sector counts and any read errors. Unreadable regions silently limit every conclusion drawn from them, and the limitation belongs in the report rather than in a deposition answer.
• Both hash values, at acquisition and again at verification, rather than a sentence in the report asserting that they matched.
• For each load-bearing finding, the second method used to confirm it and whether the two agreed. A disagreement that was investigated and explained is stronger than a finding nobody checked.
• What the tool flagged that the examiner reviewed and set aside. Filtering is an analytical decision and it leaves no trace unless someone records it.

How should the question be answered under oath?

The question arrives in one of two shapes. “What is the error rate of the software you used?” and “So you cannot tell this court how often your method is wrong?” The first is answerable. The second is a trap only if the witness accepts its premise that the method is the software.

One asymmetry is worth naming. Stating a tool’s limitations before the other side does costs a sentence in the report and removes the only surprise available on cross. Waiting to be asked converts the same sentence into a concession, and concessions are what a Rule 702 motion is written from.

Frequently asked questions

Related reading

• NIST CFTT validation and admissibility

  What a CFTT test report establishes about a named tool version, and the four things it does not establish about the examination that used it.

• The Daubert factors applied to forensic method

  All four factors translated for a digital forensic examination, with the rulings where each point was decided.

• Is MD5 still defensible for forensic verification?

  The most common misuse of a real cryptographic result in a digital-evidence dispute, and the answer that is already on the acquisition record.

• Rebutting an opposing forensic expert

  Where the tool's answer became the expert's answer — parser disagreement, carved data, and the material to request before drafting.

• The Daubert Docket

  Rulings on tool reliability among them, filterable by the ground the challenge was brought on.

Attorney advertising / expert services. General information about evidence law and forensic practice, not legal advice, and not a substitute for checking the rules and case law of your own forum.