Medallion Architecture: Design Decisions and Anti-Patterns

We've now built a complete Bronze → Silver → Gold pipeline, hands-on. In this final lecture of the chapter, let's step back and cover the questions and mistakes you'll actually run into on real production pipelines — the kind of judgment that separates someone who's followed a tutorial from someone who can genuinely own a Medallion architecture in production.

Design Decisions: Questions You Will Face on Every Real Pipeline

Design decisions — questions you will face on every real pipeline

When should you add a layer?

Add a layer when the transformation logic or the audience is different enough that mixing them creates confusion or coupling.

In other words, don't add layers just because Medallion has three — add one when you genuinely have a distinct transformation step or a distinct consumer that deserves its own boundary. Forcing everything into exactly three layers, or splintering into five layers "just in case," are both mistakes in different directions.

When should you collapse layers?

Almost never in production.

It might be tempting, especially early on, to skip Silver and go straight from Bronze to Gold "to save time." Resist this. The whole value of the three-layer contract comes from each layer doing its one job — collapsing them reintroduces exactly the problems Medallion was built to solve.

What goes in Silver versus Gold?

If the logic is about correctness, it belongs in Silver. If the logic is about answering a specific question, it belongs in Gold.

This is the cleanest possible summary of the lesson from the Gold layer lecture: deduplication, type casting, null validation — that's correctness, and it's Silver's job. "Only completed orders count as revenue" — that's a specific business question, and it's Gold's job.

How many Gold tables should you have?

One per distinct business question. If two consumers need the same answer, that's one Gold table. If they need different answers, that's two Gold tables.

This is a genuinely useful rule of thumb for avoiding both under-building (everyone hacking around one overloaded Gold table) and over-building (needless duplication of the same metric).

Should I materialize Gold, or use a view?

Materialize if query frequency is high or the join is expensive. Use a view if freshness matters more than speed.

Exactly the decision framework from the previous lecture, restated as a quick, memorable rule.

Anti-Patterns: What Will Cause a Production Incident

Knowing the rules is one thing. Knowing exactly how teams break them — in ways that cause real, painful incidents — is what actually prevents those incidents. Here are four genuine anti-patterns:

Anti-patterns — what will cause a production incident

1. Business Logic Creep in Bronze

What it looks like: a 90-day filter quietly gets added at ingestion time — maybe to save storage, maybe just to "keep things tidy."

What goes wrong: a compliance audit comes six months later, asking for data older than 90 days. The data is gone. There's no way to recover it, because it was never allowed into Bronze in the first place.

The fix: move all filters to Silver or Gold — never to Bronze. Bronze's entire value proposition is being the complete, unfiltered historical record. The moment you filter at ingestion, you've permanently traded away your ability to answer questions you didn't anticipate.

2. Silver as a Staging Area

What it looks like: Silver is written, used to build Gold, and then immediately deleted — treated as disposable scratch space rather than a real layer.

What goes wrong: Gold needs to be rebuilt (a schema change, a bug fix, a new business requirement) — and now you have to reload everything from Bronze, redoing all the deduplication and type-casting work, because Silver was never meant to persist.

The fix: Silver is persistent. It is your data of record — clean, validated, and stable enough that Gold can always be rebuilt from it directly, without ever needing to touch Bronze again.

3. Gold Table Sprawl

What it looks like: over time, the organization accumulates 47 Gold tables, each computing something almost the same metric, each defined slightly differently by whoever built it.

What goes wrong: no one knows which one is authoritative. Two dashboards show two different "revenue" numbers, and nobody can say with confidence which is correct — or why they differ.

The fix: govern Gold with Unity Catalog. Enforce one table per business metric, with clear owners, descriptions, and lineage tracked for each — exactly the governance capabilities we covered in the Unity Catalog chapter, now applied specifically to keeping Gold under control as an organization scales.

4. Undocumented Layer Contracts

What it looks like: a Silver transformation quietly excludes certain records (say, via a null filter) — with no comment, no explanation, nothing written down about why.

What goes wrong: six months later, nobody remembers why that filter exists. Was it intentional? Is it a bug? No one can say — and now no one dares touch it, for fear of breaking something they don't understand.

The fix: COMMENT ON TABLE (and on individual transformation logic) at creation time. This is a simple, low-cost habit with an outsized payoff: if you don't document it then, it never gets done. Nobody goes back later to add documentation to code that already works.

Why This Lecture Matters

Notice something about all four anti-patterns: none of them are syntax errors. Every one of these pipelines runs successfully. The code executes, the tables get built, the dashboards render numbers. The failure isn't technical — it's architectural, and it often doesn't surface until months later, in the form of a compliance gap, an expensive rebuild, a trust-destroying dashboard discrepancy, or a change nobody's willing to make.

This is exactly the kind of judgment that distinguishes a data engineer who can follow a Medallion pattern from one who can be trusted to own one in a real production environment.

Summary

That wraps up the Medallion Architecture chapter — from the concept, through hands-on Bronze/Silver/Gold implementation, to the design judgment that keeps a real production pipeline maintainable for years, not just working on day one.

See you again. Keep learning, and keep growing!