What is Medallion Architecture

Welcome to a new chapter of this course. In this chapter, we'll cover Medallion Architecture — what it is, why it exists, and how each layer fits together to build a reliable, production-grade data pipeline.

What Do We Do in Data Engineering?

At its core, data engineering is about taking data from a source, turning it into files, and then processing those files into tables that people can actually query and use.

But here's a problem that shows up quickly in real projects: what happens when you have multiple consumers, each needing the data processed differently?

The naive approach — multiple independent processes

In this naive setup, the same source files get picked up by three completely separate processes, each producing its own tables. On the surface this works — but it creates real problems as your project grows:

• Duplicated effort. Each process re-implements its own cleaning, validation, and transformation logic — even if much of that logic is identical across all three.
• No shared, trustworthy raw copy. If the source data changes or disappears, you have no single reliable place to go back to.
• Inconsistency. Since each process interprets and cleans the data independently, different consumers can end up with subtly different (and sometimes contradictory) versions of "the same" data.

This is exactly the problem Medallion Architecture is designed to solve.

What Is Medallion Architecture?

Medallion Architecture organizes your data pipeline into three clearly defined layers — Bronze, Silver, and Gold — each with a specific job:

Medallion Architecture flow

1. Bronze — Raw landing zone. Data lands here exactly as it arrived from the source. No transformation, no filtering, no interpretation.
2. Silver — Clean & conformed. Data here has been cleaned, validated, and standardized — correct types, correct schema, deduplicated.
3. Gold — Business aggregates. Data here is shaped specifically for consumption — pre-aggregated, filtered for specific use cases, optimized for reporting and BI tools.

Instead of every consumer building their own pipeline straight from source files, everyone builds on top of the same shared Bronze and Silver layers — and only the final, use-case-specific shaping happens in Gold. This eliminates duplicated effort and keeps everyone working from a consistent, trustworthy foundation.

The Three-Layer Contract: What Each Layer Owns

Here's the part that really matters in practice: each layer has a strict contract — a clear list of what it's responsible for, and just as importantly, what it must never do.

The three-layer contract

Bronze

Owns:

• Raw data, exactly as received.
• Pipeline metadata (things like _source, _file, _ts — tracking where and when each record came from).
• Complete history — every record that has ever arrived.

Never:

• Transform or cast types.
• Filter or reject records.
• Update existing rows.

Silver

Owns:

• Correct types and schema.
• Deduplicated records.
• Null-validated, quality-guaranteed rows.

Never:

• Answer business questions.
• Apply use-case-specific filters.
• Pre-aggregate data.

Gold

Owns:

• Pre-aggregated business metrics.
• Use-case-specific filters.
• Data read-optimized for BI tools.

Never:

• Store raw or uncleaned data.
• Define correctness rules (that's Silver's job).
• Serve as a source of truth for other Gold tables.

Sticking to this contract is what keeps a Medallion pipeline maintainable as it grows — every layer has one job, and no layer quietly takes on responsibilities that belong to another.

Where Medallion Pipelines Break in Production

Understanding the contract is one thing — but knowing exactly how teams violate it in real projects is what actually prevents painful production incidents. Here are four common failure modes:

Where medallion pipelines break in production

1. Skipping Bronze

The mistake: treating raw files in S3 (or another storage bucket) as if they are your Bronze layer.

Why it breaks: S3 is not Bronze. Bronze must be a queryable Delta table with history. Without it, you lose the ability to replay your pipeline without going all the way back to the original source — which may have already changed, moved, or been deleted by the time you need to reprocess.

2. Business Logic in Bronze

The mistake: applying filters or business rules at ingestion time, before data even reaches Bronze.

Why it breaks: Every filter applied at ingestion is an irreversible decision. If that rule turns out to be wrong, you can't undo it without a full reload from the source — data that may no longer be filterable the same way, or may not even exist anymore.

3. Gold Without Silver

The mistake: aggregating business metrics directly from raw (Bronze) data, skipping Silver entirely.

Why it breaks: If you aggregate directly on raw data, you lose the ability to explain where any number came from. When someone asks "why does this dashboard show 4,200 orders instead of 4,150?", you need Silver's clean, validated, traceable data in between — without it, debugging becomes effectively impossible.

4. Treating Gold as Immutable

The mistake: assuming Gold tables are stable and don't need to be touched once built.

Why it breaks: Gold is a materialization of Silver — meaning it's a derived, computed representation of Silver's data. If Silver changes (a bug fix, a late-arriving correction, a schema update), Gold must be rebuilt to reflect that change. Teams that forget this end up shipping stale dashboards — numbers that look confident but are quietly out of date.

Summary

Medallion Architecture isn't just a naming convention for three folders — it's a discipline. Each layer has a job, a contract, and a set of things it must never do. Respecting that contract is what separates a Medallion pipeline that scales cleanly from one that becomes an unmaintainable mess.

See you again. Keep learning, and keep growing. Thank you.