Efficiencies through data hub centralization

The challenge

A regulated financial services firm with nothing central in place. Eleven upstream systems and external providers, each with its own delivery method and cadence, and no single place where a figure could be traced to source.

Maturity was improving but ways of working and requirements were unstable.

The solution

The build ran in a fixed sequence. Environments, identity and the promotion path first. Then an untouched landing area. Then governance, before any consumer access. Then one use case end to end on real data with real access controls. Then gradual expansion of data scope.

The data model started with the definition of data domains. The conformed models built for the first domain did not survive requirements that were still changing. The lesson is to either commit models to how the organization is shaped, or defer structure until the organization knows what it wants.

Data quality controls came from the same build. Provider schema changes rarely arrive as an alert. Usually it is a load that succeeded with a column of nulls, noticed weeks later, or a failed job due to a different schema than normally expected. Additional controls such as schema checks, and null-rate and distribution deltas against the preceding loads, help but do not catch everything. File-level checks catch more incidents than row-level rules. Examples of controls include row count against a trailing average, latest business date present, and comparison against the previous delivery.

For actual breaks, the policy is to fail for anything feeding a reported number and quarantine row-level problems, with the rejected row and its reason retained, a named owner for the queue, and a replay path once the provider fixes the file.

The outcome

The firm went from nothing central to one governed place where a figure can be traced back to its source, used by several teams across multiple processes, with the same numbers everywhere.
All of it on a single, securely managed backbone.