About the Course

Welcome to Databricks Data Engineering with AWS.

This course is designed to teach you everything you need to become a data engineer using the Databricks platform on AWS cloud. It is well structured, and it takes you step by step from the basics to a real, complete project.

Before you start, let me quickly walk you through what you will learn.

The course has two phases:

1. Phase 1 — Foundation. Here you learn the Databricks platform itself, and all the tools it gives you for data engineering.
2. Phase 2 — Capstone Project. Here you take everything you learned in Phase 1 and use it to build one complete, real project — from start to finish. You will build it in a dev environment, move it to UAT for testing, and finally deploy it to production. This includes packaging, deployment, and a full CI/CD pipeline using GitHub.

Let's look at each phase in more detail.

Phase 1: Learning the Databricks Platform

In Phase 1, you build your foundation. You learn the Databricks tool, and everything it offers for data engineering.

We start with the basics:

• Workspace setup and Unity Catalog governance. There is a full chapter just on Unity Catalog, so you understand all its features properly.
• Delta Lake internals. Another full chapter, where you learn how Delta Lake gives you ACID transactions, time travel, and constraints.

Once the basics are clear, we move to building solutions:

• Medallion architecture. First, I explain what it is, and why it matters.
• Then, we build it by hand — using plain Spark code only, no Databricks tools. This helps you understand what is really happening underneath.
• After that, we build the same thing again, but this time using Databricks Lakeflow Declarative Pipelines.

Next, we go deep into Databricks Lakeflow. This is a big part of the course, and we cover every component:

• Lakeflow Connect — its own separate chapter. Here you learn how to bring data in from different places:
  • SaaS sources
  • Relational databases, using CDC (Change Data Capture)
  • Autoloader
• Spark Declarative Pipelines on the Databricks platform.
• Lakeflow Jobs — the last piece of Lakeflow. You learn job DAGs, retries, control flow, the REST API, and the CLI. Every feature Lakeflow gives you for jobs, we cover it.

By the end of Phase 1, you will know the Databricks platform well. This is your foundation.

Phase 2: The Capstone Project — StepRite

Now comes the fun part. In Phase 2, we put everything from Phase 1 into practice by building one complete project, from beginning to end.

The project is built for a company called StepRite — imagine it as a mid-size online footwear retailer.

Here is the setup: StepRite gives us five source systems. Data flows in from these five sources, and our job is to prepare the gold layer for the business. The business needs answers on:

• Daily revenue
• Customer 360
• Product performance
• Funnel analysis
• Fulfillment health

In this part, we design the Medallion architecture together. Every decision — the kind of decision a real architect has to make on a real project — I explain to you, and why we make it.

Going beyond just building — doing it the right way

Once the project itself is built, we don't stop there. We add things that most data engineering courses skip:

• Unit testing and integration testing for our transformation logic.
• Packaging the whole project using the Databricks Asset Bundle.
• Setting up source control in GitHub.
• Building a full CI/CD pipeline with GitHub Actions — one that runs tests, deploys to UAT, and once UAT testing is approved, deploys to production too.

So yes — this is a lot to learn. And by the end, you won't just know Databricks. You will know how to build, test, and ship a real data engineering project, the way it's done in the industry.

Quick Summary

I can't wait for you to learn all of this and master data engineering with Databricks on AWS.

See you in the course. Keep learning, and keep growing!