When tools aren’t enough: Why understanding AI evidence metrics is now a courtroom requirement

Key insights:

• • • The history of discovery is a history of matching tools to tasks — From paper review to keyword search to machine learning to generative AI, each innovation succeeded by assigning the right level of capability to the right kind of problem.

    • Adoption has outpaced understanding — AI discovery tools are now widely used, but the statistical literacy needed to defend their use in court has not kept pace.

    • Courts reward preparation, not adoption — Judges generally don’t weigh in on which tool a law firm uses; rather, they weigh in when counsel cannot explain or defend that choice.

---

Document review has always been organized around a basic question: Who, or what, is best suited to handle a given volume and type of material? In the paper era, that question was answered with people. Large cases were staffed with tiers of reviewers, working through boxes of documents. The model was labor-intensive but logical. More documents required more reviewers, and the work was distributed according to experience level, with junior reviewers handling first-pass relevance calls and more experienced attorneys handling privilege determinations and quality control.

The widespread shift to electronic discovery in the early 2000s introduced keyword search as an intermediate tool. Search terms allowed reviewers to narrow enormous document populations before human eyes even glanced at them, but the approach had limits as well. A search for “ice cream”, for example, would miss a document that said “gelato,” and a poorly constructed term list could either bury reviewers in false positives or miss documents entirely.

The next major shift came with the introduction of supervised machine learning in the late 2000s and early 2010s. This was a different kind of tool entirely. Rather than searching for fixed terms, the system learned from examples. A reviewer would mark documents as relevant or not relevant, and the system would apply that pattern across the rest of the collection, prioritizing documents most likely to matter. This was the first point at which legal discovery began to resemble a partnership between human judgment and statistical inference, rather than a purely manual or purely mechanical process.

Bringing GenAI in

The most recent shift, beginning around 2022, introduced large language models and generative AI (GenAI) into the same workflow. These tools do not simply sort or rank documents — they can summarize them, answer questions about them in natural language, and construct chronologies or timelines across a collection.

Each of these innovations did not in fact replace the one before it so much as they added a new layer of capability. Indeed, the central challenge in every era has been the same: The important determination is knowing which tool, or which level of human or machine capability, is appropriate for a given task.

---

For more on this, check out the recent “Beyond the Bench” video podcast, featuring Dr. Maura Grossman

---

This is where the analogy to legal staffing becomes useful. A first-year associate and a senior attorney are not interchangeable, because they are suited to different kinds of judgment calls. The same logic increasingly applies to AI tools. A general-purpose AI model is not the same as a fiduciary-grade system built and validated for a specific legal task. Choosing a narrowly designed, purpose-built tool over a general one is the AI equivalent of choosing an senior attorney over a law school graduate for a task that requires accountability and demonstrated reliability, not just general competence.

The literacy gap that follows adoption

As these tools have become more capable, the legal profession’s ability to evaluate them has not necessarily kept pace. Evaluating AI discovery tools, both before they are adopted and after they generate output, requires an understanding of statistical concepts such as recall, precision, confidence intervals, and margin of error. In fact, these are not concepts most lawyers were trained to work with, and many rely heavily on vendors to supply and interpret these metrics rather than developing the capacity to do so themselves. The risk is a quiet one: figures or citations that should raise concern can go unexamined simply because the people reviewing them do not know what to look for.

This gap matters because it shapes how discovery disputes are litigated. If one party argues that another’s recall rate is too low — meaning that a significant share of relevant documents was not produced — both sides need a working understanding of how that figure was calculated and what it means in order to argue the point credibly. The legal and technical questions in such disputes cannot be fully separated.

What courts actually expect

Discovery is designed to operate between parties, with judges generally staying out of methodology decisions entirely. Judicial involvement in AI-driven discovery is therefore a signal, not a routine occurrence. In fact, courts usually step in under two circumstances: i) a dispute over whether a chosen method is adequate; or ii) a timing problem severe enough to threaten the case schedule. Both situations typically indicate that something has already broken down. A low recall rate that prompts a motion or a delay serious enough to draw judicial questions, for example, suggests a deeper flaw in how the review was designed or executed and is sometimes serious enough that it can call an entire production into question rather than just the documents in dispute.

The common thread in these failures is governance, not technology. A capable AI tool used without a clear validation process, defined oversight roles, or documented standards for measuring output is no more reliable than an undertrained reviewer left unsupervised. Most of the disputes that escalate to a judge can often be traced back to a preventable gap: whether no one verified the tool’s metrics, no one understood what the numbers meant, or no one assigned responsibility for catching errors before production.

Those law firms that build proper AI governance into their discovery process, have clear protocols, defined accountability, and have a working understanding of how their tools are validated, are far less likely to need a judge to resolve what should have been caught internally.

The history of document review, at its core, is a history of matching capability to task, whether that capability comes from a person or a machine. As AI tools take on more of the discovery process, the skill that increasingly distinguishes effective use from risky exposure is not technical operation, but the ability to select the right tool for the task, understand what its outputs mean, and recognize those instances in which human review remains essential.

---

To learn more about how courts should approach AI and other advanced technology, check out the Thomson Reuters Institute’s Responsible AI Use for Courts