An insurance and reinsurance carrier that needed to safely and responsibly scale AI proofs of concept

The challenge

The carrier was running AI proofs of concept in three places: underwriting policies, approving or rejecting claims, and quoting new business. It wanted to roll them out and needed reassurance on its Responsible AI risks and controls. Nobody could say which controls were already in place, which were missing, or what had to be true before a PoC could scale beyond initial scope.

The three did not sit under the same rules. Pricing and risk assessment carry obligations that claims decisioning does not, so a single checklist would have been wrong in two directions at once, over-controlling one use case and under-controlling another.

The solution

We built the assessment framework around three obligation and risk profiles rather than one, each control traceable to a recognized standard such as the NY DFS standards and the NIST AI RMF, with insurance-specific controls added on top.

Rather than accept the carrier's own list of where AI was in use, we reconciled three inventories: what the business said it had, what vendor contracts showed it had bought (how those providers were guaranteeing control and risk management), and what the data flows actually did. The gaps between them were the first finding. Some use cases had three names depending on who was asked, and no formal owners.

Each of the more than 120 controls was then scored on four points: not observed, weak, medium, strong. Not observed means no evidence was found for the specific control, not that the control was missing. That distinction kept the conversation with control owners constructive.

What the review found

The weak and not-observed ratings were not scattered. They clustered in four areas: explainability, documentation, traceability, and control of model drift. Three of those four come back to the same underlying problem, which is that the system could not show its work after the fact.

The models and architecture could be explained. The results could not. The score produced by the model was written to a claim field and overwritten each time the claim was re-scored, so there was no durable decision record over time and no way to reconstruct the reasoning behind any given decision. Outputs were not versioned to the model's version and therefore not reproducible over time.

The factors behind an output score were being lost as well. The model returned them, but somewhere in the dataflow that field was dropped, so the one piece of evidence that would have explained a decision never reached the golden record.

Drift was the fourth cluster, and the carrier believed it was covered. Monitoring existed and watched the data going in. Nothing watched what was coming out: approval, decline and referral rates, or override rate and direction.

Bias and fairness were tested across all three use cases. Direct use of protected characteristics was ruled out quickly. Proxies were more challenging, and here the testing ran into the explainability finding rather than standing apart from it. The absence of proxy effects cannot be demonstrated without knowing what drove a given decision, and a population-level fairness result does not answer an individual complaint.

The standard response to a finding like this is that the model only recommends and a human decides. One measure settles that: override rate crossed with time on task. An override rate in the low single digits, with under 20 seconds spent per review, means the human review control is not truly performing.

The outcome

The carrier came out of the review knowing which controls it had and which it still needed, use case by use case. Whether a PoC could scale safely became a list of named gaps with effort and impact attached, allowing the business to make informed decisions on rollout plans.

Not every recommendation landed. Model owners rejected storing per-decision reasoning, implementing output versioning and ensuring reproducible results. This would have put the effort on them while producing no model performance benefit. Those recommendations were narrowed to the decisions that impact customers directly rather than interim steps and estimates.

Dozens of recommendations in total, which the carrier sequenced against its own capacity.