Customer Orders Analytics ETL Pipeline
A production-grade end-to-end ETL pipeline built on Databricks following Medallion Architecture (Bronze → Silver → Gold). Processes two datasets daily — customers and orders — through a 13-task DAG with parallel execution, a 4-check data quality gate, metadata-driven audit logging, and automated email alerts. Produces 9 Delta tables ready for business intelligence consumption.
13 with parallel execution
Pipeline Tasks
~3m 35s end-to-end
Avg Runtime
9 produced per run
Delta Tables
13 with parallel execution
Pipeline Tasks
~3m 35s end-to-end
Avg Runtime
Screenshots & Visuals Coming Soon
We are currently preparing live showcase images for this repository. In the meantime, explore the complete source code and architecture on GitHub.
codeVisit GitHub RepositoryOverview
A production-grade end-to-end Customer Orders Analytics ETL Pipeline built on Databricks using Medallion Architecture (Bronze → Silver → Gold). The pipeline processes two datasets daily — customers and orders — through a 13-task DAG with parallel execution, a 4-check data quality gate, metadata-driven audit logging, and automated email alerts on success or failure. It produces 9 Delta Lake tables ready for business intelligence consumption, runs in approximately 3 minutes 35 seconds on Databricks serverless compute, and is orchestrated entirely via Databricks Workflows.
Technical Architecture
All configuration (catalog name, schema name, all table paths, raw file paths, null rate thresholds) lives exclusively in 00_Config.py. Every other notebook imports from Config. Switching environments (dev → staging → prod) requires changing exactly two lines: CATALOG and SCHEMA. All table paths resolve automatically.
00_Config.py
Central config — single source of truth for all paths & table names
00_Start_Task.py
Pre-flight validation — confirms raw files exist before any task runs
01_Bronze_Customer/Orders
Raw CSV ingestion → bronze Delta tables with provenance metadata
02_Silver_Customer/Orders
Clean · type cast · standardise · flag invalid rows
03_Data_Quality_Check.py
4-check gate — fails fast, skips Gold if any check fails
04_Customer_Order_Mart.py
LEFT JOIN silver tables → customer_order_mart
05/06_Gold ETL notebooks
Country aggregation · revenue by category · 4 combined aggregations
07_Pipeline_Config.py
Metadata registry — config_id per entity
08_Audit_Log.py
Append-only row count audit per table per run
09_Email_Alert.py
Gmail SMTP via Databricks Secret Scope — fires on ALL_DONE
Medallion Architecture (Bronze → Silver → Gold)
Bronze: raw data preserved as-is with _ingested_at and _source_file columns added. Silver: cleaned, typed, standardised data — bad rows get _is_valid=False flag (never deleted). Gold: business-ready aggregations. Data flows forward only — Bronze data is never modified after ingestion. This guarantees full raw data preservation and clean separation of concerns across all layers.
4-Check Data Quality Gate
Before any Gold task runs, 03_Data_Quality_Check.py validates: (1) silver_customers and silver_orders are non-empty, (2) no null customer_id or order_id survived Silver, (3) total_amount null rate < 15% (configurable in 00_Config), (4) every customer_id in silver_orders exists in silver_customers. If any check fails, the notebook raises ValueError — Databricks marks all downstream Gold tasks as Skipped. Email Alert still fires (ALL_DONE dependency) with the failure details.
Metadata-Driven Audit Logging
pipeline_config table registers 8 entities with a unique config_id (1–8) covering Bronze, Silver, Mart, and Gold layers. 08_Audit_Log.py reads this registry dynamically using a JOIN — it never has a hardcoded table name. Adding a new dataset to the pipeline requires only a new row in pipeline_config. The audit log grows by one row per entity per run and is append-only — Delta Lake ACID guarantees no data loss on concurrent writes.
Databricks Workflows Orchestration
The 13-task DAG uses Databricks Workflows with explicit task dependencies. Bronze tasks run in parallel after Pipeline_Config. Silver tasks run in parallel after their respective Bronze tasks. Gold tasks run in parallel after Data_Quality_Check. Audit_Log runs after all Gold tasks converge. Email_Alert has an ALL_DONE dependency — it fires whether the pipeline succeeded or failed, ensuring the team is always notified.
Performance Benchmarks
DELTA TABLES
9 produced
PIPELINE TASKS
13 (DAG)
AVG RUNTIME
~3m 35s
DATA QUALITY CHECKS
4 gates
Results & Impact
Production-Grade Data Pipeline
Full Medallion Architecture with a 4-check quality gate, metadata-driven audit logging, and automated email alerting — the same pattern used in enterprise data engineering teams at scale.
9 Delta Tables in ~3.5 Minutes
From two raw CSVs to nine business-ready Delta tables with full provenance, type safety, and referential integrity — produced in under 4 minutes on Databricks serverless compute.
Self-Documenting Audit Trail
Append-only pipeline_audit_log with row counts per table per run — every execution is permanently recorded, queryable by date, and never overwritten.
“Raw CSV to Gold analytics in 3 minutes 35 seconds. Production-grade. Every run.”
Up next — NutriPump Artisanal