Managing AI models’ opacity and risk management challenges

Key insights:

• • • Opacity challenge — AI models operate fundamentally differently than traditional models. Unlike linear, traceable calculations, AI develops its own inferential logic that model owners often cannot fully explain or predict.

    • Third-party dependency risk — Most traditional financial institutions use foundational models from external providers rather than building proprietary ones in-house. This adds another opacity layer that makes traditional validation and monitoring nearly impossible.

    • Regulatory and trust implications — Regulators worldwide are demanding transparency and control despite these limitations. The inability to explain AI decisions undermines customer trust, complicates compliance, and creates governance gaps.

---

The challenge for financial institutions around developing customer-facing or internal models in the AI age may be simple to understand, but it’s not easy to solve. Financial institutions develop models to enhance their decision-making, improve financial reporting, and ensure regulatory compliance; and these models often are used across various banking and financial services operations, including credit scoring, loan approval, asset-liability management, and stress testing.

Traditional models — for which existing model risk management was written — often operated in a predictable, linear fashion. A model user could enter inputs, trace calculations, validate assumptions, and forecast outputs with relative confidence. These are in stark contrast to some applications of AI models, particularly those using deep learning. Often, AI model users may not be able to predict its outputs or precisely explain the model’s inferences.

The third-party complication

Here’s where things get even more complex. Most financial institutions don’t build their AI models from scratch; instead, they’re leveraging foundational models from companies like OpenAI, Anthropic, and Google. These large language models (LLMs) serve as the backbone that can be configured for everything from customer service chatbots to risk assessments.

This creates a new dimension of opacity. Banks aren’t just dealing with models they can’t fully explain; they’re utilizing models they didn’t originally build and don’t wholly control. The original training data, architecture, and parameters all remain proprietary to the model providers.

The model risk management implications are numerous. How do you validate a foundational model when you don’t have access to its training data? How do you ensure it won’t produce biased outputs when you can’t examine how it infers its data? How do you monitor for model drift when the foundational builder might update the model without notice? Traditional vendor risk frameworks weren’t designed for this level of dependency on opaque, constantly evolving systems.

When traditional risk management fails

Traditional model risk management relies on three components: initial validation, ongoing monitoring, and the ability to challenge model assumptions. Third-party foundational AI models may disrupt all three.

Initial validation becomes problematic when you’re validating a system you can only observe from the outside. Unlike traditional statistical models built on explicit assumptions, AI models develop their own inferential logic through training, which isn’t always visible.

---

Banks aren’t just dealing with models they can’t fully explain; they’re utilizing models they didn’t originally build and don’t wholly control.

---

Ongoing monitoring faces similar challenges. If an institution is relying on a foundational model like OpenAI’s GPT or Anthropic’s Claude as the basis for their own AI application, the institution is subject to the foundational model’s updates. A model that performed reliably last month might behave differently today due to changes the institution didn’t execute; the assumptions present in each version may not be readily measurable.

Further, government regulators are beginning to implement more detailed guidelines specifically targeting AI models. Financial institutions must demonstrate transparency and control over complex systems, including those they source from third parties. In mid-2024, for example, the Monetary Authority of Singapore issued guidance on AI model risk management; and now similar initiatives are emerging globally, from the United States’ Federal Reserve and Canada’s Office of Financial Institutions, to the European Union’s AI Act. However, just as fast as AI models update, global regulatory oversight and momentum can pivot near immediately.

Real-world consequences and the search for solutions

The stakes extend beyond regulatory compliance. When a model generates outputs that are understood only by a team at an external company, operational risks can cascade. For example, customer service representatives often need to explain why a fraud system flagged a transaction; or loan officers must be able to provide specific reasons why a credit model rejected an application — and black box AI makes these basic requirements nearly impossible.

The trust deficit affects everyone. Customers denied services without clear explanations lose faith, and regulators struggle to verify compliance. Internal audit teams may not offer confidence when models are proprietary third-party systems, and board members face governance questions they can’t adequately answer.

The industry is responding with various approaches. Some institutions are demanding greater transparency from AI providers, negotiating for access to model documentation and performance metrics. Others are building testing frameworks to validate third-party models through extensive input-output analysis.

Techniques like SHAP and LIME attempt to illuminate black box decisions by approximating how models weight different factors. Some institutions are adopting hybrid approaches, combining simpler, interpretable models with complex foundational models to balance performance with transparency.

---

Financial institutions must demonstrate transparency and control over complex systems, including those they source from third parties.

---

These solutions involve trade-offs, however; chiefly that more interpretable models may sacrifice predictive power. Post-hoc explanation techniques provide approximations, not perfect transparency. The tools for managing third-party AI model risk are still maturing, even as deployment accelerates.

What needs to happen now

Financial institutions must build explainability and control mechanisms into their AI journeys from the start. This may require cross-functional teams of data scientists, risk managers, compliance officers, and vendor management specialists who can negotiate appropriate terms with foundational AI providers.

Institutions also need comprehensive governance frameworks that address the unique challenges of third-party foundational models. This could include enhanced vendor due diligence, continuous monitoring, contractual provisions for model transparency and update notifications, and a willingness to forgo some AI capabilities when risks can’t be adequately managed.

Still, the fundamental tension remains: AI’s power comes partly from its ability to identify trends at scale, and currently, operating in ways we don’t fully understand. When third-party providers are thrown into the mix, predictability and control become even more tenuous. Institutions must leverage the benefits of foundational models while acknowledging what remains unknown and outside their direct control.

If attained, this comprehension can be a strategic driver. Institutions that can harness third-party AI’s power while maintaining genuine oversight will gain a competitive advantage. Those that don’t may face serious consequences if black boxes from third parties produce outcomes they can neither explain, predict, nor defend. In an industry where trust and compliance are paramount, it is crucial for financial institutions to truly comprehend AI-associated risks.

---

You can find out more about how financial institutions and other organizations manage their risk here