The lawyer is still accountable. The AI system acting on her behalf is not. That gap is no longer theoretical.
After convening the first meeting of the Trust in AI Alliance, it is clear this mismatch is emerging as one of the biggest barriers to enterprise AI deployment.
As AI systems move from answering questions to taking action inside professional workflows, a fundamental mismatch is emerging. Execution shifts to the system. Responsibility still sits with the human.
In agentic systems, that model is being reconfigured, but there is still no clear answer to a critical question: how does a human maintain accountability as more of the work is executed by the system?
That question was at the center of the inaugural convening of the Trust in AI Alliance, a group bringing together leaders across model development, infrastructure, and enterprise AI deployment, where participants from OpenAI, Google, Anthropic, AWS, and Thomson Reuters discussed what trustworthy agentic systems require in practice.
A clear theme emerged: AI capability is accelerating faster than accountability.
Most systems today are not designed for that standard.
The Shift No One Is Talking About
In the first wave of AI, the defining question was whether a system could produce a correct answer. That is no longer enough.
As AI systems take on multi-step tasks across real workflows, the question is shifting from accuracy to accountability.
As Michael Gerstenhaber, Vice President of Product Management at Google, said during the discussion: “Delegating agency to a synthetic agent implies trust. The more you delegate, the more you need observability, tracing, and audit. It is not one feature. It is defense in depth.”
In traditional professional environments, accountability is clear. Humans determine relevance, review source material, verify outputs, and take responsibility for outcomes. In agentic systems, that model is evolving.
Retrieval is automated. Context is lost across steps. Outputs appear grounded in source material without preserving fidelity. Tools execute beyond the user’s visibility.
As Frank Schilder, Senior Principal Scientist at Thomson Reuters, noted: “When we move to an agentic workflow, we automate steps that professionals used to perform manually and that introduces new risks: Context can be silently dropped. Source fidelity can become fragile. Maintaining clear accountability becomes more complex.”
These are not edge cases. They are structural risks. We are automating the work, but not accountability.
If You Can’t Inspect It, You Can’t Trust It
In regulated industries, trust has never meant blind confidence. It has always meant the ability to verify. That standard is now colliding with how many AI systems operate.
Accuracy drives experimentation. Inspection determines adoption.
If a system cannot show its work, it cannot be trusted in high-stakes environments.
As Gayle McElvain, Head of TR Labs at Thomson Reuters, put it: “Errors create liability. For many professionals, trust means ‘trust but verify.’ That means building AI systems where verification is built in.”
Across the discussion, several consistent priorities emerged around what trustworthy systems must provide:
-
- Step-by-step auditability
- Traceable reasoning and inspectable tool use
- Durable logs and process artifacts
- Clear, persistent provenance
This is not a feature. It is infrastructure.
Trust Breaks When Source Integrity Breaks
In knowledge-based professions, trust depends on the integrity of source material.
Agentic systems introduce new failure modes. They may paraphrase where precision is required. They may surface outdated information. They may blur the boundary between authoritative sources and generated reasoning.
These are not cosmetic issues. A single altered word in a statute can change its meaning. A misapplied version of a regulation can create real consequences.
As Zach Brock, Engineering Lead at OpenAI, described: “We are moving toward agents that share durable scratch spaces. Citations, version identifiers, and hashes of source material can travel through a workflow without being compressed away.”
That level of persistence is not a technical detail. It is what makes accountability possible.
Without it, professionals cannot trace how an answer was constructed or verify whether it reflects the correct source at the correct point in time. Without it, accountability breaks.
Accountability does not emerge automatically from more capable systems. It must be explicitly defined.
As Byron Cook, Director of Automated Reasoning at AWS, said: “With AI, some of those socio-technical mechanisms go away. We have to define the dividing line between behaviors we accept and those we do not—and enforce that symbolically. Without that, accountability cannot be maintained as systems take on more of the work.”
This Is a Systems Problem
Much of today’s AI development is optimized for performance benchmarks. But in real-world environments, performance is only part of the equation.
As Scott White, Head of Product, Enterprise at Anthropic, noted: “Benchmarks measure whether a model can do the task. Enterprises are asking a bigger question: will the system around it hold up in the environments where the work actually happens? A trustworthy agent requires the model, the boundaries around it, and the record of what it did. Getting all three right is what turns AI from a powerful tool into a system enterprises can trust with important work. That’s what will drive the next wave of adoption.”
Trustworthy systems must be designed to operate safely under pressure, with clear boundaries and strong safeguards.
That requires:
-
- Clear separation between system instructions and external content
- Built-in safeguards against prompt injection and data leakage
- Continuous monitoring and testing
- Audit trails aligned with regulatory expectations
Agentic AI is not just a model challenge. It is a governance challenge.
The Next Phase of AI
We are entering a new phase of AI adoption, one defined not by experimentation, but by deployment inside real workflows.
The industry is shifting from outputs to systems, from benchmarks to reliability, and from capability to accountability.
But this shift will not happen automatically. It requires new standards for auditability, clearer approaches to provenance, and systems designed to preserve truth and responsibility across every step of a workflow. These are solvable problems—but only if accountability is designed into the system from the start.
The organizations that solve this will define the next generation of AI.
In high-stakes domains, trust is not optional.
It is not a feature. It is the product.
The Trust in AI Alliance was announced in January to bring together leaders across the AI ecosystem to advance practical standards for accountability, transparency, and trust in AI systems. The group will continue to meet regularly, with select insights from those discussions shared publicly.
