A credible answer and an inaccurate one can look remarkably similar. On the surface, both look factual. For anyone who’s staked their professional reputation on the answer, though, the difference is everything.
That’s the gap we opened our second Trust in AI Alliance session with.
The first session looked at model accuracy. This time, participants from OpenAI, Google, AWS, and Thomson Reuters pushed a layer deeper. How do we know when an AI system is reliable, and how do we get there?
The Gap Between the Lab and the Real World
Evaluation still has ground to cover, especially once a real person is the one asking. A strong benchmark score is usually where development starts, not proof that a system is ready for real work. A model can perform well under controlled conditions and still fall apart once someone is using it in the middle of an actual job.
Andrew Bean, a Lead Research Scientist on our foundational research team, opened the session with a definition of trust as “the willingness of a party to be vulnerable to how another party acts.” It reframes the question from “is the model good?” to “do we understand it well enough to rely on it?”
A study he led, published in Nature Medicine, makes the point concrete. Language models identified relevant medical conditions correctly more than 90 percent of the time under controlled testing. In a study of roughly 1,300 people using the models, that number dropped to 35–40 percent. The only variable that changed was the human interaction.
That’s the gap. Passing a test is one thing. Holding up when a real person, under real conditions, is asking, that’s another.
Trust Requires Learning as Systems Are Used
Most evaluation methods can catch outright fabrication, but they can miss a conclusion that goes slightly beyond what the source says.
Frank Schilder, Senior Principal Scientist at Thomson Reuters, put it plainly, “because it’s so fluently written and nicely formatted, then the guards are down.” The better these systems sound, the more work shifts to evaluation. A model failing occasionally isn’t the real concern, not knowing how or why it fails is.
Vasant Rajamani, who leads OpenAI’s enterprise work, framed the challenge as “iterative deployment and improving quality over time.” In this way, you can build systems that track when they underperform to focus on continual improvement.
Evaluation can’t stop at launch. The signal must feedback continuously, weighted by risk and tied to the policies an organization already has, so the system keeps improving in production rather than freezing at launch.
A Word on Regulation
Evaluation doesn’t happen in isolation. It sits inside a regulatory environment that’s moving quickly, particularly around data and security. As questions around data retention become more prominent, a fair question follows. How much responsibility should concentrate with a small number of model providers, versus the enterprises and professionals building on top of them?
There’s no settled answer yet. What is clear is that the regulatory environment needs to keep pace with how fast capability is moving. The goal here isn’t less oversight. It’s oversight that reflects how these systems are actually built and used, spread across model providers, application builders, and the organizations deploying them, rather than resting on any single layer. In that environment, evaluation is not a ‘once and done’ exercise, but a tool to represent real performance that builds trust from the end user and to give confidence in the decisions made with AI.
Oversight Has to Match the Risk
This is why Thomson Reuters holds its products to a Fiduciary-Grade AI™ standard. Professionals should never simply accept what an AI system gives them, but how closely they check should match the stakes.
As the discussion emphasised, reviews can’t become a formality in place of real scrutiny. The discipline is knowing which moments need a closer look.
That challenge grows as more workflows run without a person checking every step. That’s the point of automation, but someone still decided a human wasn’t needed there, and that decision must hold up on its own. Responsibility doesn’t disappear when a human steps back, in fact, it moves earlier, into the design choice itself.
Madhu Parthasarathy, GM of Amazon Bedrock AgentCore, described what that looks like in practice. Hand-writing a policy for every domain doesn’t scale, so the approach is to derive policies from real interactions, verify them with automated reasoning, and bring a person in to validate the ones that carry the highest consequence cases. Those become part of layered, inherited policy, a company-wide rule that every agent must follow, with teams adding their own on top.
That solves what the system is allowed to do. It doesn’t solve who, or what, is trusted to do it. Michael Gristenhuber, VP of Product for Gemini Enterprise at Google, put it this way, “we don’t really have a system for credentialing the operator and not the operation yet.” People earn trust through licenses, track records, and peer review built up over time; AI systems have no equivalent yet.
Until they do, the safeguard must live inside the system itself, and nobody owns the answer to where that threshold sits, whether that’s the model provider, the organization building on it, or the government.
Where the Interface Comes In
None of these lives in one place. The model must know where it’s shaky. Someone must decide how hard to check it, and that decision must hold up before anything goes wrong. The tool itself must tell a person what kind of answer they’re looking at. Miss any one of those, and the whole chain is only as good as the part nobody checked.
That’s what we’ll dig into next. How an interface can help someone spot a good answer versus one that just sounds good, and how asking for a second opinion builds trust too.
No single model or company will solve this on its own. It takes trial and error across all of it, and the discipline to keep asking where the next gap is. For AI built for professionals, that’s not a feature. It’s the job.
