Introduction to Data Engineering

Welcome to Mastering Databricks for Data Engineers.

In this course, we will learn Databricks — its capabilities, and how to use the Databricks platform along with Apache Spark to solve data engineering problems.

But before we jump into Databricks, let's take a step back. Let's first understand data engineering itself — what a data engineer actually does, and what happens in a data engineering project. Once we are on the same page about this, we will move on to learning how to do it using Databricks and Spark.

So, let's start with the basics: what is data engineering?

Who Interacts With Enterprise Data?

Take a look at this diagram (from Google Cloud's documentation). It shows all the people who interact with an enterprise's data, directly or indirectly.

Data Engineers – Reference Architecture

These people fall into three groups, based on what they do:

1. Business Data Owners

These are the people who build and manage operational systems — the systems your day-to-day business runs on. Think banking apps, e-commerce apps, OTT platforms, IoT applications. These are the applications your customers interact with every day.

Why are they called "data owners"? Because these are the applications that capture and collect data straight from your customers. They collect it, store it, and hold on to it. That's why they own it.

2. Data Consumers

These are the people responsible for optimizing, growing, and monitoring the business, day to day.

• Optimize the business — for example, a fraud prevention system.
• Grow the business — for example, a recommendation system that drives more sales.
• Monitor the business — for example, sales and revenue reports that track KPIs.

Why are they called "consumers"? Because to build any of these systems, they need data — the same data that the business data owners are collecting and holding on to. They consume the enterprise's data to build data-driven decisions and applications.

3. Data Engineering Team

So now we have a gap: data owners are sitting on the data, and data consumers need that data to build their systems. Who connects the two?

That's us. That's the data engineering team.

Here's what we do:

1. We collect data from the systems owned by business data owners.
2. We transform and process that data.
3. We facilitate consumption — we hand it over in the shape the data consumers need.

In short: we sit in the middle. We connect data owners and data consumers, by moving the right data, at the right time, at the right frequency. That, in one line, is what a data engineering project is all about.

The Three Functions of Data Engineering

Now that we know why data engineering exists, let's break down what a data engineer actually does. Every data engineering project can be split into three functions:

1. Data Integration / Data Ingestion

This is where it all starts. We collect data from the source systems — because that's where the data owners keep it. We bring that data into our own possession, so we can work with it.

2. Data Engineering / Data Processing

This is the biggest, most complex part of the job — so big that we just call the whole field "data engineering" after it. Some people call it data transformation. This is where most of the real work happens: cleaning, shaping, and preparing the data according to what our consumers need.

3. Data Modeling and Consumption

Once the data is ready, we need our consumers to actually use it. That usually means building data marts or analytical stores, modeled the way our consumers want to consume it. This last function is about modeling the data correctly, and making it easy to connect to and consume.

So, in short — ingest, process, model. That's the three-function view of any data engineering project.

Three Approaches to Data Engineering

On top of these three functions, there are three approaches you'll come across when implementing a data engineering project:

1. Batch processing — data is processed in scheduled chunks (e.g., every day, every hour).
2. Real-time stream processing — data is processed instantly, as it arrives.
3. Near real-time stream processing — somewhere in between — small delay, but still very fast.

You'll pick one of these three, depending on what the project needs.

The Reference Architecture: Lakehouse Medallion Architecture

We now understand what data engineering is, where we fit in, what we do, and the different ways we do it. But to actually build all this, we need an architecture — a blueprint for how to structure our platform.

This is where the Lakehouse Medallion Architecture comes in. Here's the diagram:

Lakehouse Medallion Architecture

Let's walk through what it means, left to right.

Step 1 — Data Ingestion / Data Integration

This is the first function we talked about. We pick a suitable tool (based on our source system) to collect the data. We can do this in one of two ways:

• Batch — collect data at regular intervals (daily, hourly, every 30 minutes, etc.)
• Streaming — collect data continuously, as it's produced

The architecture supports both — and you can even mix them. Some data can come in batch, some in streaming, in the same project.

Step 2 — The Three Layers: Bronze, Silver, Gold

After ingestion, the architecture recommends organizing everything into layers. At a minimum, you need three layers:

• Bronze layer — stores the raw data, exactly as it was ingested. No changes.
• Silver layer — this is where we read from bronze, and apply our data engineering work: cleaning, transforming, and improving quality.
• Gold layer — the final layer, where data is shaped and modeled specifically for consumption — ready for reporting, apps, or ML models.

So the flow is simple: ingest the data, then move it forward step by step — bronze → silver → gold.

And just like ingestion, this processing can happen in batch mode, near real-time mode, or full real-time streaming mode. The architecture supports all three.

What's Next

We now understand the concept — the Lakehouse Medallion Architecture. But how do we actually build it? What technology do we use?

That's exactly what this course is about. We are going to implement this Lakehouse Medallion Architecture using Apache Spark and Databricks.

Let's get started in the next lecture.

Keep learning, and keep growing!