A specialist data practice for regulated workloads.

Decide the shape of the data estate before you platform it.

Most "data platform" projects fail because the architecture was decided implicitly, in the gap between business strategy and engineering capacity. We make that architecture explicit, debated, and decided — before any platform engineering starts.

Architecture patterns we use

Data-product thinking

We treat every analytical dataset as a data product — with an owner, a contract, an SLA, and a lifecycle:

• Owner. A named team (not a person) accountable for the dataset's accuracy, freshness, and availability.
• Contract. Schema, semantics, allowed values, freshness expectation, breaking-change policy — defined and reviewed before consumers depend on it.
• SLA. Documented freshness, availability, and quality targets.
• Discoverability. Surfaced in the catalog with description, lineage, and example queries.
• Versioning. Breaking changes go through a deprecation path; consumers get notice.
• Deprecation. Datasets that no consumer reads are deprecated, archived, deleted — on a schedule, not a whim.

Build vs buy

We make explicit decisions about which capabilities to self-host, which to take as managed services, and which to consume as SaaS:

• Self-host where data-sovereignty requires it, or where operating-cost dynamics favour it at scale.
• Managed cloud services (BigQuery, Snowflake, Redshift, Databricks, Synapse) where the data-residency posture and operating overhead trade-offs work out.
• Specialist SaaS (Fivetran, dbt Cloud, Hightouch, Atlan, Monte Carlo) for capability areas where building or self-hosting is not where the engineering team should spend its time.

Lakehouse-first. Self-hosted where it earns its keep.

Our default data platform is a lakehouse — open table formats on object storage, with Spark and SQL query engines on top, running on the same OpenShift fleet as the rest of the platform. Managed cloud warehouses are absolutely on the table where they fit; we don't insist on self-hosting for its own sake.

Lakehouse stack

Managed cloud warehouses

When sovereignty, scale, or operating overhead points to a managed warehouse, we deliver the same engineering discipline on top:

• Snowflake. Account / role architecture, RBAC patterns, virtual-warehouse sizing, cost attribution, masking and row-access policies.
• Google BigQuery. Dataset and project layout, slot reservations, column-level security, BigQuery-ML integration where it fits.
• Amazon Redshift / RA3. Managed storage, workload management, materialised views, integration with AWS Lake Formation.
• Databricks. Workspace design, Unity Catalog, photon engine tuning, ML and analytics on the same surface.
• Azure Synapse / Fabric. Dedicated vs serverless pools, integration with Azure data services, Fabric workloads.

Why we lean lakehouse for regulated workloads

• Open formats. Iceberg and Delta on object storage means data is portable. You are not contractually locked into a single vendor's query engine.
• Cost decoupling. Storage cost decouples from compute cost; query engines can be scaled or replaced independently.
• Co-location with platform. Running on the same OpenShift fleet as your other workloads simplifies identity, networking, security posture, and observability.
• Disconnected-friendly. Self-hosted lakehouse stacks run cleanly in disconnected and air-gapped environments where managed cloud is not an option.

Events as the spine of regulated systems.

For regulated workloads — payment authorisation, fraud detection, claims intake, network telemetry — "real-time" is not a nice-to-have. We design and operate the streaming spine that those workloads ride on, with explicit guarantees about ordering, durability, and delivery semantics.

Streaming platforms

Stream processing

• Apache Flink. Stateful stream processing with strong exactly-once guarantees, event-time windowing, large-state support. Default for stateful streaming jobs.
• Spark Structured Streaming. Where the team already operates Spark and the streaming workload fits the micro-batch model.
• Kafka Streams & ksqlDB. For Kafka-native, simpler stream transformations.
• Debezium for CDC. Change-data-capture from operational databases into the streaming spine, with snapshot and incremental modes.

Schema, contracts, and evolution

A streaming platform without schema discipline becomes a swamp very quickly. We design schema management into the platform from day one:

• Schema registry. Confluent Schema Registry, Apicurio, or equivalent — with required schemas per topic and a compatibility policy.
• Schema formats. Avro for compact wire format with backward-compatible evolution; Protobuf where ecosystem fit demands it; JSON Schema for human-readable contracts.
• Compatibility policy. Explicit backward / forward / full compatibility per topic, with breaking-change deprecation paths.
• Topic contracts. Owner, semantics, retention, partitioning, expected throughput, allowed consumers — documented in the catalog.

Operational discipline

• Dead-letter handling. Every consumer has a DLQ pattern with documented retry, alerting, and reprocessing path. Silent loss is treated as a defect.
• Delivery semantics. Each topic and consumer documents its at-most-once / at-least-once / exactly-once posture and the trade-offs that justify it.
• Observability. Lag per consumer group, throughput per topic, schema-violation rate, DLQ depth — all in the same dashboards as the rest of the platform.
• Disaster recovery. Multi-region replication topology (MirrorMaker 2, Confluent Replicator, or equivalent), documented RPO/RTO, regularly drilled.

Governance as engineering — not as a committee.

Data governance fails when it is run as a slide deck overseen by a steering committee. We embed governance into the same engineering surface as the rest of the platform — the catalog updates automatically, the lineage is captured at the source, the data contracts run as CI checks, and the audit evidence is a side effect of normal operation.

Catalog & discoverability

Lineage

• OpenLineage. Standard for emitting lineage events from Spark, Airflow, dbt, and other tools into the catalog.
• Column-level lineage via dbt and supported catalogs — not just table-to-table arrows.
• Cross-system lineage from CDC source through streaming → lake → warehouse → BI tool — tracing a regulatory report all the way back to its operational origin.
• Lineage as audit evidence. A regulator question about "where did this number come from?" answerable in minutes, not weeks.

Data contracts & quality

• Schema and semantics contracts defined per data product, versioned in Git, reviewed in PR.
• Quality tests. dbt tests, Great Expectations, Soda — running in the same pipeline as the transformation, failing the pipeline if the contract breaks.
• Freshness SLAs. Each data product has a freshness target; misses produce alerts and incident tickets.
• Anomaly detection. Monte Carlo, custom heuristics, or platform-native tooling to catch statistical regressions before downstream consumers do.
• Quality dashboards per data product, showing SLA compliance, top failure modes, and active incidents.

Master Data Management (MDM)

• Domain-by-domain scoping. Customer, product, account, counterparty — we MDM one domain at a time, not boil-the-ocean.
• Golden-record design. Match-merge logic, deterministic and probabilistic patterns, with the survivor rules documented and reviewed.
• Source-system reconciliation. Reconciliation reports per source, with mismatches routed to data stewards.
• Stewardship workflow. Web UI for stewards to review proposed matches, approve merges, and resolve conflicts.
• Operational MDM vs analytical MDM. We are explicit about whether the golden record drives operational systems (writes back) or is purely analytical.

Privacy & access control

• PII detection and classification across the data estate, refreshed on schema changes.
• Row-level and column-level security tied to your identity provider, not bolted on per-tool.
• Masking and tokenisation for PII in non-production environments.
• Purpose-based access controls for regulatory regimes (GDPR purpose-limitation, sectoral consent regimes).
• Audit logging of every access event, retained per regulatory requirement.

A semantic layer your reports can trust.

Most BI estates accumulate hundreds of reports that agree on the headlines and disagree on the numbers. The fix is rarely "buy a new BI tool" — it is putting a semantic layer between the warehouse and the BI tool so that "revenue", "active customer", and "delinquency" mean one thing.

Semantic layer

• dbt as the transformation and semantic backbone for analytical models — tests, documentation, lineage, version control, all in one engineering practice.
• Cube, MetricFlow, or LookML as the consumer-facing semantic layer, depending on the BI tool and the contracting model.
• SQLMesh for organisations needing first-class virtual environments and time-travel-friendly modelling.
• Metrics-as-code. Every metric defined once, versioned in Git, peer-reviewed, with explicit ownership.

BI tooling

Self-service governance

Self-service analytics fails when the boundary between governed and ungoverned queries is unclear. We design that boundary explicitly:

• Certified marts. Datasets that have passed quality, lineage, and ownership checks. Tagged in the catalog and the BI tool.
• Sandbox tier. Where analysts can explore, model, and prototype — without their work appearing in board-level dashboards by accident.
• Promotion path. Documented route for moving a sandbox dataset into the certified tier — review, tests, ownership, SLA.
• Deprecation path. Reports nobody reads get archived; orphaned datasets get deleted.

Reporting that matters

• Operational reporting tied to the source-of-record system.
• Regulatory reporting with end-to-end lineage from the report back to operational data.
• Embedded analytics in customer-facing applications.
• Executive scorecards that the board actually uses — with explicit definitions and a single source of truth.

AI rides on the governed data layer — or it doesn't ride at all.

Most enterprise AI programmes stall at the data layer. The lesson is consistent across customers: AI workloads built on shadow pipelines fail internal audit; AI workloads built on the governed analytical surface ship. We build the data layer so AI is a first-class consumer.

Feature stores

• Feast as the default open-source feature store on top of the lakehouse — offline store on Iceberg/Delta, online store on Redis or low-latency KV, registry in Git.
• Platform-native stores — Databricks Feature Store, Vertex AI Feature Store, SageMaker Feature Store — where the data estate is anchored on a single platform.
• Feature lineage. Every feature traceable back to its source data, with version history.
• Online/offline consistency. Documented guarantees on the relationship between training-time and serving-time feature values.
• Feature SLAs. Freshness and availability targets per feature, monitored in production.

Training-data pipelines

• Reproducible dataset versions. Every training run pinned to a specific dataset version, with signed manifests.
• PII handling. Tokenisation, masking, or full removal at ingestion. Differential privacy where the use case allows.
• Consent and purpose. Training-data inclusion tied to documented consent and purpose-limitation policies.
• Labelled-data pipelines with human-in-the-loop tooling (Label Studio, Argilla, vendor platforms), version control, inter-annotator agreement.
• Synthetic data for cases where real data is scarce, sensitive, or restricted — with documented validation against real-data behaviour.

Retrieval grounding for LLMs

RAG is, fundamentally, a data problem. The model is interchangeable; the retrieval is what makes the answer right. We build the retrieval surface as a data product:

• Source-document pipelines. Ingestion from authoritative sources with provenance, version, and last-modified tracking.
• Chunking strategies calibrated to your content: section-based for legal/policy documents, sentence-based for support content, hybrid for mixed corpora.
• Embedding pipelines with embedding-model version, batch vs streaming refresh, re-embedding triggers on document update.
• Vector store choice driven by latency, scale, sovereignty — not by ecosystem hype.
• Retrieval evaluation with a labelled eval set, recall@k metrics, blocking regressions on retrieval changes.
• Audit trail. Every retrieved chunk surfaced to the user with citation, retained for audit.

Cross-cutting

This tab overlaps with our AI practice on inference, agentic patterns, MLOps, and AI governance. The data work is the substrate; the AI work is what runs on top.