Apache Spark to Data Engineering Platform

Welcome back. In this lecture, let's understand two things:

1. What is Apache Spark — and what is it not.
2. Why do we need something like Databricks (or a similar platform) to build enterprise-grade data engineering solutions and implement the Medallion architecture.

Let's start with Apache Spark.

What is Apache Spark?

In simple words: Apache Spark is an engine for running data engineering, stream processing, and machine learning workloads on a distributed cluster of computers.

It gives you all of this through a set of interfaces:

• Spark SQL — ANSI SQL compliant
• DataFrame APIs — for batch processing
• Structured Streaming — for stream processing
• GraphX — for graph processing
• MLlib — for machine learning

All of this, in a single, unified framework.

Spark is used everywhere. Thousands of companies run on it — including 80% of Fortune 500 companies.

The Spark Stack

Let's understand how Spark is put together, layer by layer.

What is Apache Spark – A Unified Framework

At the bottom — storage. Spark itself doesn't store data. It reads and writes from distributed storage systems like HDFS, S3, ADLS, or Google Cloud Storage.

Above storage — the compute cluster and resource manager. Spark is a framework that runs on a cluster of computers, not a single machine. To run on a cluster, it needs a resource manager — Spark supports several: YARN, Standalone, Kubernetes, Mesos.

Above that — the Spark Framework itself, sitting on top of the cluster and resource manager.

On top of the framework — Spark Core APIs. These are also called RDD APIs, and they're available in four languages: Scala, Java, Python, and R. These are the lowest-level APIs — not something we typically use directly.

On top of Spark Core — the APIs we actually use day to day:

• Spark SQL / DataFrame APIs
• Structured Streaming
• MLlib (machine learning)
• GraphX (graph computation)

These higher-level APIs are simpler and more powerful, and this is what most data engineers work with.

Why is Spark So Popular?

A few reasons:

• Abstraction. Spark hides the complexity of distributed processing. You write a simple SQL query or a DataFrame operation — you don't need to think about how the data gets split and processed in parallel across a cluster. Spark handles that for you, behind the scenes.
• Easy to use. You can start with something as simple as Spark SQL, or use APIs in the language you're comfortable with.
• Unified platform. One platform gives you SQL, DataFrames, batch processing, streaming, machine learning, and graph processing — all together.
• Open source. This is a big one. It's free, and it's community-driven.
• Wide ecosystem. Because it's open source, companies across the industry have built integrations and tools around it. Spark connects easily with a huge range of systems.

That's why Apache Spark is the most popular engine for data processing — and especially for big data engineering.

What Apache Spark is Not

Now here's the important part. Spark is powerful, but it is not a complete solution by itself. Building an enterprise-grade data engineering solution needs more than just Spark.

Here are five critical things that Apache Spark, on its own, does not provide:

1. No storage layer. Spark has no storage of its own. It always needs to be paired with something else — HDFS, S3, ADLS, etc.
2. No ACID transactions. ACID means Atomicity, Consistency, Isolation, and Durability — a core requirement for any reliable data system. Spark doesn't guarantee this on its own. You need to bring in another tool to add ACID capability on top of Spark.
3. No robust, centralized metadata catalog. Spark has a small, basic catalog, but nothing close to a proper enterprise-grade metadata system.
4. No cluster management. Spark cannot create, destroy, or manage clusters by itself. That's the job of a separate cluster manager.
5. Limited automation tooling. Enterprise projects need a lot of automation — APIs, SDKs, CLI tools. Spark doesn't offer a strong set of these out of the box.

So here's the simple takeaway: Apache Spark is the most popular big data engineering engine, but it's not complete on its own. It needs to be paired with additional tools and capabilities to become a true enterprise-grade solution.

This is exactly where Spark platforms come in.

Spark Platforms

Let's look at some of the popular platforms built around Spark:

• Cloudera Hadoop Platform — the most popular on-premise Hadoop platform, usable as a Spark platform.
• Amazon EMR — Amazon's Spark offering.
• Azure HDInsight — Microsoft's Spark offering.
• Google Dataproc — Google Cloud's Spark offering.
• Databricks — the platform this course is built around.

What's the difference between them?

Cloudera, Amazon EMR, Azure HDInsight, and Google Dataproc are all, behind the scenes, based on Hadoop technology. Even the cloud ones (EMR, HDInsight, Dataproc) launch a Hadoop cluster behind the scenes and use YARN as their resource manager. Since Spark can run on Hadoop, these platforms work as Spark platforms too — but Hadoop is the foundation underneath.

Databricks is different. It is a cloud-native, pure Spark platform — completely free of Hadoop. It does not run a Hadoop cluster behind the scenes, and it does not use YARN. It's built from the ground up, on the cloud, for the cloud, using Spark and cloud-native technology directly.

That's what makes Databricks today's most popular and most powerful Spark platform for building enterprise-grade data engineering solutions on the cloud. One thing to note: as of now, Databricks is a cloud-only platform — it's not available for on-premise deployments.

Summary

Now you understand what Apache Spark is, what it isn't, and why a platform like Databricks matters. This course is all about Databricks — so let's go learn the platform itself.

See you in the next lecture. Keep learning, and keep growing!