What Was Holding the Platform Back
A fast-scaling direct-to-consumer (DTC) meal-kit and grocery subscription company had built its analytics foundation on Snowflake. As the business expanded across multiple geographies, the platform began to create structural blockers to growth:
Runaway Infrastructure Costs
Snowflake credit consumption grew super-linearly with subscriber count. High-frequency ingestion, unoptimised analyst queries, and always-on development warehouses pushed the annual platform bill to the point where unit economics on new customer cohorts were materially impacted.
No Real-Time Personalisation
Recipe recommendations operated on nightly batch refreshes, leaving preference signals many hours stale. Attempts to introduce streaming via Snowpipe produced latencies in the 30–90 second range — far too slow for a real-time customer selection portal.
Fragmented ML Lifecycle
Data scientists exported results to S3, trained models in SageMaker, and re-imported predictions back into Snowflake for BI consumption. This three-system handoff made reproducibility audits painful and typically added several weeks to each model-deployment cycle.
Semi-Structured Data Left Out
Clickstream events, IoT cold-chain sensor data, and customer support transcripts lived in S3 but were largely excluded from analytics because Snowflake's VARIANT handling added significant overhead at the volumes involved.
Governance & Lineage Gaps
Data assets were spread across Snowflake, S3, SageMaker, and a partially decommissioned Redshift environment. Proving end-to-end lineage for GDPR and SOC 2 Type II audits required manual effort and was error-prone.
How We Approached the Migration
Lucent Innovation held a 10-week evaluation — including proofs-of-concept on Databricks, Snowflake with Snowpark/Cortex, and a Redshift rebuild — before recommending Databricks Lakehouse Architecture. We designed a phased Lift · Evolve · Optimise migration to ensure zero analytical downtime throughout.
Foundation & Schema Migration (Months 1–3)
Reviewed all existing data sources, workflows, and pipelines. Re-targeted dbt models to the Databricks adapter. Deployed Delta Lake on AWS S3 using Terraform. Enabled bi-directional dual-write between Snowflake and Databricks with automated reconciliation jobs — any divergence above 0.01% on business-critical KPI tables triggered a Slack alert and rollback.
Streaming & ML Pipeline Cutover (Months 4–6)
Migrated streaming ingestion from Snowpipe to Auto Loader on Structured Streaming (Kafka → Delta). Moved ML training from SageMaker to Databricks ML Runtime with MLflow. Established a Databricks Feature Store (Unity Catalog) and moved the recommendation engine to Databricks Model Serving. Snowflake retained as read-only fallback throughout.
Optimisation & Decommission (Months 7–9)
Photon engine tuning on SQL warehouses. Z-ordering and liquid clustering applied to high-cardinality fact tables. Batch jobs moved to spot/preemptible instances. Unity Catalog governance policies fully enforced. Snowflake contract wound down on schedule.
Medallion Architecture
The Final Platform Stack
| Platform | Databricks Lakehouse |
| Storage Format | Delta Lake on AWS S3 (open Parquet; time-travel enabled) |
| Streaming Ingestion | Apache Kafka + Databricks Auto Loader (Structured Streaming) |
| Batch Ingestion | AWS DMS + Auto Loader (CDC from PostgreSQL / MySQL) |
| Transformation | dbt Core (Databricks adapter) + PySpark |
| Orchestration | Databricks Workflows (DAG-based, cluster lifecycle managed) |
| ML Training | Databricks ML Runtime + MLflow (GPU clusters for deep learning) |
| Feature Store | Databricks Feature Store + DynamoDB (offline + online serving) |
| Model Serving | Databricks Model Serving — serverless REST endpoints |
| Analytics & Reporting | Databricks SQL Serverless + Power BI / Tableau |
| Infrastructure as Code | Terraform (hybrid AWS + Azure deployment) |
| Governance | Unity Catalog — unified data, model, and feature lineage |
| Security | Okta SSO + Unity Catalog RBAC + VPC — SOC 2 Type II compliant |
