Project 02 · Lakehouse + orchestration

Telecom Billing Lakehouse

Bronze, Silver, and Gold tiers built over synthetic Call Detail Records. Airflow orchestrates ingestion onto Iceberg-on-S3, Great Expectations specifies contracts at the boundary, and dbt models analyst-ready marts on top.

View on GitHub Explore the tiers Architecture deep-dive

0.53s

raw to Gold, 49,998 rows

~94k rows/sec, single process

41/41

dbt tests passing

8 models, bronze to gold

Bronze contract expectations

types, nulls, ranges, MSISDN regex

unit tests passing

generator + transform

Measured 2026-07 on the local demo path (synthetic data, seed 42) — reproduce via the repo's Results section.

Bronze · rawSilver · cleansedGold · martsClick a band to explore that tier

Tier drill-down

Bronze, Silver, Gold — what changes between them

Click a tier above or use the tabs below to see the schema, sample rows, transformations, and data-quality checks at each stage of the medallion.

Bronze · Raw landings

~ 1.2 B rows · synthetic

Append-only landing zone. We keep raw payloads as they arrived so any downstream issue is replayable.

Sample rows

raw_payload	ingested_at	source_file	partition
{"caller":"+1-214-555-0142","callee":"+1-415-555-0188",…	2026-04-01T08:15:01Z	cdr_2026_04_01_08.json.gz	dt=2026-04-01/hr=08
{"caller":"+1-512-555-0190","callee":"+44-20-7946-0958"…	2026-04-01T08:15:02Z	cdr_2026_04_01_08.json.gz	dt=2026-04-01/hr=08
{"caller":"+1-214-555-0173","callee":"+1-718-555-0119",…	2026-04-01T08:15:13Z	cdr_2026_04_01_08.json.gz	dt=2026-04-01/hr=08
{"caller":"+1-415-555-0188","callee":"+1-214-555-0142",…	2026-04-01T08:15:45Z	cdr_2026_04_01_08.json.gz	dt=2026-04-01/hr=08
{"caller":"+1-303-555-0166","callee":"+1-303-555-0166",…	2026-04-01T08:16:04Z	cdr_2026_04_01_08.json.gz	dt=2026-04-01/hr=08

Schema

raw_payloadstring (JSON)
ingested_attimestamp
source_filestring
schema_versionstring
ingest_partitiondate

Transformations applied

Schema-on-read JSON parse
Append to ingest partition
Capture source file + ingested_at

Data-quality checks

Schema validation via Great Expectations
Null-rate <= 2% on caller/callee
Source-file checksum match

Orchestration

Daily Airflow DAG

A linear pipeline that generates synthetic CDR data, lands it on Bronze, validates with Great Expectations, normalizes into Silver, builds Gold marts via dbt, and notifies on completion.

Airflow DAG · daily lakehouse pipeline

6 tasks · runs nightly at 02:00 UTC · LocalExecutor

queuedrunningsuccess

Data quality

Scorecard across the four pillars

Completeness, validity, uniqueness, and freshness — measured by the dbt + Great Expectations test surface and surfaced into a simple scorecard.

Synthetic demo data

Completeness

% of records with all required fields populated.

Validity

% of records that pass type, range, and regex checks.

Uniqueness

% of records with no duplicate primary key.

Freshness

% of partitions arriving within their SLA window.

Data quality scorecard - illustrative placeholder values.
Metric	Value (%)	Description
Completeness	99.4	% of records with all required fields populated.
Validity	98.7	% of records that pass type, range, and regex checks.
Uniqueness	100.0	% of records with no duplicate primary key.
Freshness	99.9	% of partitions arriving within their SLA window.

dbt lineage

From sources to marts

Hover or focus a model to see its source file. The graph mirrors the dbt project's actual model layout — 8 models, 41 passing tests.

dbt lineage · sources to gold

Hover or focus a model to see its source file.

Sources (Bronze)Silver modelsGold marts

Architecture

From producer to BI

The full pipeline from synthetic generator to BI consumers. The engineering narrative below covers the trade-offs reviewers ask about.

flowchart LR
        G[Synthetic CDR generator] -->|parquet| R[(MinIO/S3 raw zone)]
        R -->|Airflow ingest DAG| BR[(Iceberg Bronze)]
        BR -->|Airflow transform DAG
GE checks| SI[(Iceberg Silver)]
        SI -->|dbt run| GO[(Gold marts
revenue_by_market, arpu_monthly, churn_signals)]
        GO --> BI[BI / consumers]

Engineering narrative

Problem, judgment calls, and measured results

The write-up a senior reviewer would ask for — why this architecture, what was traded away, and what the numbers actually say.

Problem

Telecom billing teams sit on enormous CDR (Call Detail Record) volumes and need a single source of truth for revenue, ARPU, and churn signals. The Medallion pattern (Bronze raw, Silver normalized, Gold dimensional) keeps late-arriving data and corrections tractable while exposing clean marts to BI. This project is an end-to-end implementation: synthetic CDR generation, Airflow orchestration, Iceberg storage on S3-compatible object storage, data contracts via Great Expectations, and dbt for the analytics layer - all runnable locally with no AWS account.

Judgment calls

Why medallion over a single table? Telecom CDR volumes are huge and lossy. Keeping a Bronze append-only layer means a schema change, a parsing bug, or a vendor-side correction can be replayed without re-ingesting from upstream. Silver is where validation happens but rows are never dropped - bad rows are flagged so analysts can audit them. Gold is the only layer BI tools see.

Iceberg vs Delta. Both work. Iceberg’s hidden partitioning and partition evolution make schema migrations far less painful than rewrite-on-write tables, and it gives engine portability if a downstream consumer wants Trino or Athena instead of Spark. Delta would be a fine choice in a Spark-mostly shop; for a batch-first billing workload, Iceberg is the cleaner fit.

Great Expectations at Bronze, dbt tests at Silver/Gold. GE specifies the contract before bad data enters the lake - schema, null rates, ranges, the +1NNNNNNNNNN MSISDN regex; dbt tests catch the bad relationships afterward - uniqueness, referential integrity, accepted values. The two cover complementary failure modes. Honest caveat: the bundled demo DAG wires the GE checkpoint as a demonstrative stub, so blocking enforcement requires the full Airflow + GE stack; the 10-expectation contract itself lives in version control either way.

Measured results

The no-infra demo path is fully measured and reproducible with three commands. make demo takes 49,998 synthetic CDRs from raw parquet through Bronze, Silver, and three Gold marts in 0.53 s (~94k rows/sec, single process). make dbt-run && make dbt-test builds all 8 models and passes 41 of 41 tests - unique, not_null, accepted_values, relationships, and expression checks across every layer. The dbt project vendors local replacements for its two dbt_utils macros, so it runs completely offline with no package downloads.

What I would do differently in production

Replace the GE 0.18 API with Great Expectations 1.x (or pandera) once a stable migration path lands - and wire the checkpoint for real enforcement.
Add lineage via OpenLineage emitters on every Airflow task and dbt run.
Add cross-column contracts (e.g. roaming flag must match a roaming-eligible plan).
Replace LocalExecutor with KubernetesExecutor for scale-out.
Add an Iceberg compaction DAG to keep Bronze partition file counts bounded.

Stack

What this project uses, and why

Apache Airflow
Apache Iceberg
MinIO / S3
Great Expectations
dbt
Terraform

See the full code

The repo runs locally with no cloud account — architecture doc, tests, and reproducible measurements included.

View repository Architecture doc

All projects Next: AI-Assisted SQL Optimizer