With careful planning and execution, courts and non-profits can provide a helpful and comprehensive AI-powered resource that improves access to justice for those that need it the most
Low-income people in the United States can’t afford adequate legal help in 92% of civil matters, and the promise of AI could potentially make legal services more affordable, according to the Legal Services Corporation. In fact, several court systems and nonprofits are demonstrating this promise, a couple of which were recently highlighted in a webinar series hosted by the National Center for State Courts (NCSC) and the Thomson Reuters Institute’s AI Policy Consortium.
For example, the People’s Law School in British Columbia developed the chatbot Beagle+ to assist people with step-by-step guidance on everyday legal problems. Drew Jackson, Digital & Content Lead at the People’s Law School, led the efforts to create Beagle+ with technical assistance from Chris McGrath, Founder of Tangowork. And the Alaska Court System (ACS) and LawDroid using a grant from the NCSC to develop an AI-powered chatbot called the Alaska Virtual Assistant, or AVA. Jeannie Sato, Director of Access to Justice Services of ACS worked with Tom Martin, CEO and Founder of LawDroid, to develop the tool.
How courts can successfully experiment with AI
Jackson, McGrath, Sato, and Martin all offered their step-by-step guidance on how courts and nonprofits can experiment and use AI successfully within courts systems.
Step 1: Determine the problem
When starting a generative AI (GenAI) legal assistance project, it is crucial to first pinpoint the specific legal needs and challenges faced by your target audience. McGrath noted that he sees several common examples, including providing public access to legal information, creating internal resources like bench books for judges, and automating court document preparation.
To properly identify the problem, conduct thorough user research to understand pain points related to accessing and applying legal information. For instance, Martin suggests starting by speaking with court staff. “I think we sometimes get caught up in the excitement about wanting to throw AI at the problem and create a solution,” Martin explains. “And there are many use cases, but I think the part that’s really important is to meet with your staff, meet with everyone who’s being impacted by the burden of work, and then determine, based on that, what is the best choice.”
Taking the time upfront to clearly define the problem will help ensure that any AI solution being developed is truly meeting a demonstrated need.
Step 2: Craft a vision
Shifting from problem identification to crafting a vision for the GenAI-powered solution is crucial. The People’s Law School’s Beagle+ chatbot illustrated this well. “Begin with the end in mind,” says Jackson. “When you begin a project, keep in mind what you’re trying to achieve and what success looks like because that’s going to be different for each person.”
Jackson further described how in 2018, the initial vision was to create a chatbot capable of intelligently answering questions about consumer and debt law in British Columbia. Today, while that vision is realized, the ability of GenAI technology to adapt and improve over time necessitates a continuous and evolving vision.
Step 3: Allocate realistic resources
Assessing available resources is crucial before embarking on a GenAI project, with a realistic evaluation considering such factors as existing legal content, technological capabilities, staff expertise and capacity, and budget.
It’s important to examine the state of the organization’s existing legal information, including its documents and web pages, to determine the quality and consistency. Indeed, conflicting information across sources often can confuse GenAI models.
For staff capacity, Sato explains how the ACS started with a small team of people, which included the court administrator, the chief technology officer, a webmaster, and two to three staff attorneys, who were necessary for content review, testing, and feedback. It is not uncommon for an initial project to consume about 30% of each team member’s time.
Technological expertise is also a key consideration in resource assessment. In fact, Martins says this underscores the importance of working with a technology partner that can help navigate the different choices and options available, including the need to understand options for AI model selection, vector databases, and embedding strategies. While some may consider using large language models (LLMs)to reduce costs, the expenses for setup and maintenance often outweigh the benefits compared to using established services like OpenAI.
Financial resources are also a consideration, of course; however, it is worth noting that the cost of OpenAI tokens is often surprisingly low compared to other project expenses. For the creators of Beagle+, for example, using OpenAI’s tool has cost no more than $75 per month, according to Tangowork’s McGrath.
Courts can explore the possibilities of AI tools in tackling their specific legal challenges by experimenting within NCSC’s AI sandbox
Addressing common concerns
Our experts say that two common concerns often arise when considering the use of GenAI to solve justice gaps: one is the need for multilingual capabilities; and the second is how to handle AI-generated inaccurate information, or so-called hallucinations.
“Advanced LLMs like GPT-4 demonstrate impressive multilingual capabilities and are able to understand and respond in numerous languages on-the-fly without requiring additional training or configuration,” explains McGrath. “Multilingual support is a key advantage of modern LLMs, enabling chatbots to serve diverse populations with minimal additional development effort.”
However, hallucinations are a significant concern when using LLMs for legal applications. Fortunately, the combination of several advanced strategies can mitigate hallucinations:
-
-
- First, grounding responses in providing context through techniques like retrieval-augmented generation can help tether outputs to verified source material.
- Second, careful prompt engineering and relevancy scoring can further constrain responses.
- And finally, automated checks that compare model outputs to source documents can flag potential hallucinations.
-
At the same time, manual expert review by humans — known colloquially as human in the loop — remains crucial, even with automated safeguards in place. Therefore it is key to periodically sample responses for human verification and focus more intensive review on higher-risk conversations.
Creating a successful AI-powered chatbot for legal information requires careful consideration of the several steps cited above. By following these actions and staying up to date with the latest developments in AI technology, courts and organizations working to close the justice gap can create effective and responsible chatbots that provide valuable legal information to those who need it most.
You can register here for the upcoming NCSC webinar on March 19, which will explore the principles and practices of responsible AI implementation in court settings
