Zack GregoryArchitecture & projects
Selected architecture work
A closer look at the design decisions behind multi-ERP integration, pipeline delivery and governed enterprise reporting.
Vantage Specialty Chemicals · Platform modernization
From ERP silos to a shared lakehouse
Across 10+ ERP systems, the same customer, product or order could have several identities. I led the design and delivery of a Databricks lakehouse that preserved source history and gave finance and commercial teams a common analytical model.
Incremental ingestion retains source history. Conformed entities feed the analytical model and governed business measures.
01Sources and ingestion Capture
Use source-specific keys and load boundaries. Incremental ingestion through Auto Loader preserves raw payloads and their arrival context so a transformation can be replayed.
02Bronze Delta Retain
Keep the source record, extraction timestamp and source identity together. Raw snapshots preserve the evidence for diagnosing late or changed transactions.
03Silver Delta Conform
Standardize keys, units and currency conventions. Eighteen conformed Silver tables used source-to-enterprise bridges and Type 2 history to keep prior entity versions addressable.
04Gold enterprise model Model
Give orders and invoices separate fact grains. Join to conformed dimensions through effective-dated keys so a new classification does not rewrite historical analysis.
05Fabric and Power BI Serve
The modernization roadmap placed Fabric and OneLake downstream of the curated model. A shared Power BI semantic model centralizes measure definitions and access policy.
Engineering choice
Conform identity in Silver before aggregating in Gold. It takes more mapping work up front, but a source key stays traceable when two ERPs use it for different customers.
Delivery and governance
Unity Catalog structured access by layer and domain. Governed data products reduced manual reconciliation, with estimated annual savings of approximately $40K.
Affirma Consulting · Meta engagement
Data engineering at team scale
At Affirma, I managed the 21-person data pipeline engineering team supporting the Ray-Ban Meta and Oakley Meta smart glasses programs. The work included a centralized vendor-partnership platform built with Databricks and Python.
01Ingest Source contract
Record the source identifier, batch boundary and arrival time before transforming any rows. A stable input allows a failed run to be reproduced.
02Orchestrate Dependency gates
Release a downstream task when required inputs are ready. Partition the work by a bounded batch or time window so retries do not replay the entire estate.
03Transform Idempotent writes
Merge at the business key and the correct event grain. A repeat run should produce the same published state.
04Validate and observe Quality gate
Check key uniqueness, referential integrity and reconciliation totals. Capture rejected counts and lag alongside the batch identifier; hold publication when critical checks fail.
05Publish Consumer contract
Expose tested tables with a defined owner and refresh expectation. Backfills must pass the same checks as scheduled loads.
Delivery judgment
Bound the interface between teams at a documented data contract. A breaking schema change can then be isolated to the producer and affected consumers.
Consulting scope
I owned technical discovery and statements of work for Amazon, Microsoft and Meta engagements. The client programs I led at Affirma delivered more than $12M in aggregate measurable value.
ClubCorp · Enterprise business intelligence
One definition across 140+ clubs
Club-level extracts and different reporting conventions made comparisons difficult. As Director of Data Analytics & BI, I led the Tableau-to-Power BI migration and rebuilt the semantic and reporting layer for operations and marketing.
01Club activity Transaction grain
Capture activity at its natural grain. Keep the original club identifier so an aggregate can be reconciled to its contributing records.
02Dimensional model Shared dimensions
Relate facts to consistent club, date and activity dimensions. Establish a one-to-many path from dimension to fact and avoid fan-out when slicing by club.
03Semantic model Measure ownership
Define the numerator, denominator, reporting period and allowed filters once. Evaluate ratios from aggregate components rather than averaging club-level percentages.
04Reporting Shared consumption
Managers and executives consume the same measures at different levels of detail. Validate totals against a reference period before retiring a legacy report.
Trade-off
Centralized measures reduce drift across reports. A clear ownership and review process gives local operators a way to propose exceptions without creating a second definition.
Delivered outcome
Centralized Azure data pipelines replaced club-level extracts and reduced refresh times. A shared reporting layer supported operational and marketing analysis.
Vantage Specialty Chemicals · Financial and commercial analytics
Making orders comparable across ERPs
An order line and an invoice line answer different questions. The global data model work defined a common representation for each, with bridges for product and customer identity.
| Source record | Local quantity | Enterprise product | Standard quantity |
|---|---|---|---|
| ERP A / A-017 | 2 cases × 12 kg | Product 101 | 24 kg |
| ERP B / B-404 | 10 kg | Product 101 | 10 kg |
Both records map to Product 101. A product-specific conversion yields 34 kg across the two rows.
Modeling choice
Resolve units by product and effective date. Preserve the native amount and currency alongside the reporting amount, FX rate and conversion date.
Intended capabilities
This approach supports consolidated open orders, CRM snapshots, price-volume analysis and profitability reporting. These are use cases from the modernization roadmap.
Architecture decisions
Where the trade-offs live
A design review starts with the failure a business owner needs to detect. These choices make that conversation concrete.
01Incremental or full load Change capture
Use incremental processing when a stable cursor, late-arrival policy and replay path exist. Prefer a bounded full refresh for small sources that cannot reliably expose changes. Compare recovered counts before promotion.
02Type 1 or Type 2 Entity history
Overwrite attributes when a correction should apply to all history. Version a dimension when historical reporting must reproduce the value effective at transaction time.
03Central or local measure KPI ownership
Centralize measures with shared business meaning and an accountable owner. Allow scoped extensions for local needs; name and document their population and aggregation rules.