Welcome back to The AI Law Professor. Last time, I examined Kirkland & Ellis's $500 million platform investment and what a serious research budget signals for law firms operating at a fraction of that scale. This month I want to look at what can go wrong with AI, as two disclosures in late July showed what AI agents are capable of when the isolation around them fails.

What happened

Before an AI lab releases a model, it tests how good the model is at hacking. The usual format is a scored exercise: Here is a pretend company, a secret is hidden on its systems, go find it. The test runs inside what is supposed to be a walled-off network with no routes to the outside world.

On July 21, OpenAI reported that during one of these exercises, its models went looking for a way out. The models found a flaw nobody knew about in a piece of software the test environment relied on, used it to reach the open internet, and broke into Hugging Face, a widely used site in which AI models and datasets are shared. The models targeted Hugging Face because they calculated that its database had the answers to the test grading.

Nine days later, Anthropic published its own containment failure. It reviewed 141,006 test runs and found three cases in which a model reached the internet from a testing partner's environment and got into a real organization. In one instance, the pretend target shared a name with a real website, so the model attacked the real one and took passwords and extracted several hundred rows of live data. Then Meta, not to be outdone, disclosed that its AI model had gone rogue too.

The models followed their instructions

All the accounts describe a similar failure. Each model had been told it had no internet access, so when real systems turned up, it treated them as part of the exercise.

Consider that a new associate is told that every file he or she is given is a training exercise. If given a live client file, the same as the rest, the associate will likely treat it as a test too. AI agents work the same way. They act on what they believe about the situation, and that belief can be wrong. When your firm gives a tool access to a document management system, its authority stops only where you have built a physical limit to its access.

The security failures were unremarkable

Ordinary weaknesses let these break-ins happen: A password that was easy to guess; a web page meant for developers, left visible to the public with credentials printed on it; and fake software that acted like a Trojan horse.

That last one works because modern software is assembled from thousands of small free components pulled automatically from public libraries. A model found a setup document naming a component nobody had ever published, wrote one under that name, and waited for machines to install it. Fifteen of them did, and it collected their passwords.

Security experts have described these incidents as a preview of what routine attacks will look like within months, as open-source models put the same capability in far more hands. Any firm that has deferred basic security housekeeping on the theory that nobody would bother with a practice its size is relying on a false assumption.

Take precautions before it’s too late

To mitigate these hacker risks, law firms — and all organizations, for that matter — should take some basic protective steps. First, start with a list. Any tool that holds a password to one of your systems, browses the web on your behalf, or can save files is an AI agent, no matter what the vendor calls it. Enforce security controls that are impossible to avoid rather than instructions that an AI tool can reason its way past. Restrict which systems a tool can reach.

Then handle the basics, on a schedule for which someone is accountable. Establish two-factor authentication on every system holding client information, and ensure that no pages are reachable without a login. Have the developer tools switched off in anything live, and apply updates on a schedule that does not get forgotten. Then, monitor what the tools actually do, because logs nobody reads offer no protection at all.

Ask the same questions of your vendors' vendors. One testing firm's misconfiguration produced incidents at two separate labs. Model Rule 5.3 asks you to ensure nonlawyer assistance meets your professional obligations, and that duty does not stop at the company whose contract you signed.

Then write the response plan while nothing is happening. Who decides an incident has occurred? What does the notification clock look like under state breach statutes and your engagement terms? Does your cyber-policy cover activity your own systems started? How long are records kept of what a tool did, so that reconstructing events is even possible?

It’s your ethical obligation to disclose

Comment 8 to Model Rule 1.1 puts the risks of technology inside the duty of competence, while Rule 1.6(c) requires reasonable efforts to prevent unauthorized disclosure of client information. And ABA Formal Opinion 483 covers what to do after a breach.

We know about these recent incidents because the labs investigated and then disclosed what they found. Competitors published accounts of their own failures, called organizations that had no idea they had been compromised, and invited independent review.

Firms adopting these tools will have incidents too; but the ones that come through well will have decided in advance to look, to tell, and to write down what they learned.

Tom Martin is CEO & Founder of LawDroid, Adjunct Professor at Suffolk University Law School, and author of the forthcoming AI with Purpose: A Strategic Blueprint for Legal Transformation (Globe Law and Business), where he shares exactly how you can build your own strategic habits and assets.‍