Nov 10, 2025 | AI and product innovation

Grading our work 

The importance of trusted, high-quality content and human expertise cannot be overstated when it comes to legal AI. But what happens when that content and expertise is used not just for retrieving answers, but for evaluating the output itself? For CoCounsel, this results in faster skill building and a more meaningful human presence in the development process.  

The CoCounsel development process is anything but linear, which is exactly what you want! We build, we test, we refine, we grade, we tweak – all with the goal of creating the most reliable output in an industry where variables abound. One of the most critical steps in our development process is evaluation. During evaluation, we’re making sure that our skills and workflows are not just providing an answer, but providing the right answer.  

In order to evaluate a skill or workflow’s output, we use a comprehensive rubric  to measure completeness, accuracy, and relevance. Typically, these grading rubrics are created by a team of attorneysubject matter experts and detail what components should be present in an answer, what level of detail should be included, what underlying legal considerations should be present and accounted for, and even what best practices should be followed. CoCounsel skills are then evaluated against this rubric – not just once or twice – dozens of times with variations in prompts to determine if it measures up to what a gold standard response should look like. Even if the adjustment hinges on a tiny change, the process begins anew. All in all, an extremely thorough yet lengthy process.  

Refining evaluation through quality content 

When looking for new ways to speed up evaluation without sacrificing quality or human involvement, our CoCounsel team tapped into Practical Law for answers.  

As a testament to the depth and breadth of the nuanced resources available, our technology teams found that the detailed process and legal considerations checklists on Practical Law could be used to develop rubrics for grading AI output. Authored by our team of over 650 Practical Law editors – all with real-world legal practice experience – these checklists provide step by step instructions for completing legal tasks or tackling complex legal questions, including what laws, regulations, and best practices should be followed. And now, Practical Law checklists are also serving as a foundation of success, ensuring that CoCounsel takes the right steps, provides the right information, and connects the right dots.  

What this looks like in action 

For example, Jonathan Viray, a software engineer for CoCounsel and licensed attorney,  relies on  Practical Law to not only provide the knowledge layer for our CoCounsel skills but also to pave the way for a more streamlined testing process. The instructive content written by legal professionals, for legal professionals is now informing the way we grade our work. 

“As we’re pushing to an agentic-first experience, we’re now looking for ways to improve how we evaluate those outputs. Evaluations are very time-consuming. We have to use a lot of legal subject matter experts, and it takes a lot of time to generate the tests themselves,” Viray says of the extensive testing and evaluation process. “The Practical Law checklists are really helpful in speeding up the process in early stages of development.” Where relevant checklists exist, our tech team can use them as rubrics, cutting down on the time it takes to develop the rubric itself. 

During these early development stages, it’s imperative to get feedback on the outputs so our teams can move fast and adjust our AI to make sure we’re providing trustworthy answers. Before incorporating Practical Law checklists, this initial evaluation process could take days if the output is a simple clause or weeks if we’re evaluating multi-step, complex workflow outputs. After leveraging existing Practical Law content, our tech team was able to speed up their early iterative development by using checklists as a rubric to automatically grade CoCounsel’s work before handing it off for legal expert evaluation. “We’re speeding this process up by days or weeks,” Viray emphasizes. “This is not replacing subject matter experts. It still has to go through human feedback and actual experts to grade, but they’re able to review much faster because they have a more refined output and more tailored analysis.” 

Content that informs and refines 

We’ve said it before, and we’ll say it again: domain specialization matters. So, what does this evaluation process have to do with specialization? The answer is: everything. Yes, CoCounsel’s output is grounded in the industry-leading legal content on Westlaw and Practical Law. But we’ve taken it a step further. This content, uniquely backed by decades (upon decades) of editorial enhancements made by in-house legal experts, also serves as the guiding light for what gold-standard legal AI output can and should look like. This is content that refines our offerings and informs the way forward. No trolling the open web, no incorporation of unverified, outdated, or just plain misleading information from an armchair “legal expert” – just real subject matter experts relying on content drafted by real lawyers to make sure CoCounsel can support the work of real legal professionals at the pace the industry demands.