Chapter 11 — Why NumPy? Thinking in Arrays

Lesson 11.1 — From Loops to Arrays: Why Speed Matters

Here's a story worth sitting with for a moment. A data engineer needs to convert a column of a million prices from dollars to a discounted price — a simple calculation, applied a million times. Written as a plain Python loop, the way you've been writing code all of Week 1, it takes about 40 seconds. Written a different way — the way you're about to learn — it takes under half a second.

Same result. Same computer. Eighty times faster. That difference isn't a small optimization. At real data volumes, it's the difference between a pipeline that finishes before your coffee's ready, and one that's still running an hour later, holding up everyone downstream who's waiting on it.

The tool behind that difference is called NumPy — short for Numerical Python. It's the foundation almost the entire data engineering and data science world is built on, including the tool you'll spend most of this week with, pandas. Understanding NumPy, even just the basics, is what makes pandas make sense, instead of feeling like memorized magic.

Here's the core idea, in plain language, before we touch any code: a normal Python list holds items loosely, and doing math on it means looping through one item at a time, checking and converting types as you go — exactly what you've been doing since Chapter 5. NumPy instead stores a whole batch of numbers together, tightly packed, all guaranteed to be the same type. Because of that guarantee, it can hand the whole batch to your computer's processor at once, and let it crunch all the numbers in parallel, instead of one at a time. That's what "vectorized" means, and you'll hear that word constantly from here on — it simply means "operating on a whole batch at once, instead of looping."

You don't need to understand the hardware details to use this well. You just need to build a new instinct: whenever you're about to write a loop to do math over a batch of numbers, pause, and ask whether NumPy can do it in one line instead. That instinct is worth more than memorizing NumPy's full toolkit — the toolkit you'll pick up naturally, with practice, over this chapter and the rest of the course.

Try It Yourself: Before the next lesson, think of one calculation from Week 1 where you looped through a list doing math — the running-total pattern from Chapter 5, for instance. Keep it in mind; you'll redo it the fast way very soon.

Lesson 11.2 — Creating & Indexing NumPy Arrays

Let's get NumPy installed and write your first array. Open your terminal in VS Code:


bash
pip install numpy

Now, the array itself — NumPy's core building block:


python
import numpy as np

prices = np.array([24.99, 15.50, 89.99, 8.25])
print(prices)
print(type(prices))

Output / Note

[24.99 15.5 89.99 8.25] <class 'numpy.ndarray'>

Notice that import numpy as np line — you'll type it, exactly like that, at the top of nearly every data script you write from now on. as np gives NumPy a short nickname, purely to save typing — it's such a universal convention that every data engineer recognizes np. on sight.

An array looks a lot like a list, and behaves like one in plenty of ways — indexing works exactly the same:


python
print(prices[0])
print(prices[-1])

Output / Note

24.99 8.25

Same zero-based indexing, same negative-index shortcut, from Chapter 4. What's genuinely different starts showing up once you index with a condition instead of a position — something a plain list can't do at all:


python
print(prices[prices > 20])

Output / Note

[24.99 89.99]

Read that carefully, because it's a new shape worth understanding properly: prices > 20 doesn't just check one value — it checks every value in the array at once, and hands back a matching array of True/False results. Then wrapping that inside prices[...] uses those True/False values to pick out only the matching prices. This pattern — called a boolean mask — is genuinely one of the most useful ideas in this entire chapter, and you'll meet its pandas equivalent very soon, in Chapter 13.

Arrays also come with useful built-in summaries, ready instantly, without writing a loop:


python
print(prices.sum())
print(prices.mean())
print(prices.max())
print(prices.min())

Output / Note

138.73 34.6825 89.99 8.25

Each of those replaces what would have been a small loop with running totals back in Week 1 — now it's one word.

Try It Yourself: Create an array called row_counts with the values [4500, 0, 3200, 980, 0]. Print its .sum(), .mean(), and .max(). Then use a boolean mask to print only the values greater than 0.

Lesson 11.3 — Vectorized Operations: Math on a Whole Column at Once

Now let's put real math to work — the exact kind of calculation that opened this chapter, applied to an entire array at once, no loop required.


python
import numpy as np

prices = np.array([24.99, 15.50, 89.99, 8.25])

discounted = prices * 0.9
print(discounted)

Output / Note

[22.491 13.95 80.991 7.425]

Read prices * 0.9 like a sentence: "multiply every single value in this array by 0.9, all at once." No loop, no for price in prices, no manually building up a new list — just the math, written directly, exactly the way you'd write it on paper. This is the entire idea of vectorization, and it applies to every basic operation you'd expect:


python
quantities = np.array([3, 1, 5, 2])

totals = prices * quantities
print(totals)

print(totals.sum())

Output / Note

[ 74.97 15.5 449.95 16.5 ] 556.92

Two full arrays, multiplied against each other, element by element — the first price times the first quantity, the second times the second, and so on — all in one line. Try to picture writing that with a for loop and a running total, the way you would have in Week 1. It would work, but it would take several lines, and run dramatically slower on any real-sized dataset.

You can also combine a mask with a calculation, which starts to feel like real, useful logic:


python
prices = np.array([24.99, 15.50, 89.99, 8.25])

expensive_total = prices[prices > 20].sum()
print(expensive_total)

Output / Note

114.98

Read it in three steps, the way Python actually evaluates it: find every price over 20, keep only those, then sum what's left. Three ideas, one readable line.

One honest, practical note worth remembering: NumPy arrays are genuinely great at this kind of column-wide math, but on their own, they don't carry column names, mixed data types, or the row-and-column table shape that real datasets have. That gap is exactly what pandas fills, starting in the next chapter — and now that you understand what's underneath it, pandas is going to make a lot more sense.

Try It Yourself: Given prices = np.array([100, 250, 40, 600, 15]) and tax_rate = 0.08, calculate the total price including tax for every item in one vectorized line. Then use a boolean mask to find the total tax owed on only the items priced over 100.

Lesson 11.4 — Hands-On: Benchmark a Loop vs. a Vectorized Calculation

Let's prove the very first claim of this chapter, with your own two eyes, on your own machine. You're going to time the same calculation two different ways: a Week 1-style loop, and a vectorized NumPy calculation.

The scenario: you have 500,000 order totals, and every one needs an 8% discount applied.


python
import numpy as np
import time

order_count = 500_000
order_totals = np.random.uniform(10, 500, order_count)

np.random.uniform(10, 500, order_count) generates half a million realistic-looking random prices between 10 and 500 — a quick way to build a large, honest test dataset without needing a real file.

First, the loop-based way, exactly how you'd have written this in Week 1:


python
start = time.time()

discounted_loop = []
for total in order_totals:
    discounted_loop.append(total * 0.92)

loop_time = time.time() - start
print(f"Loop version took: {loop_time:.4f} seconds")

Output / Note

Loop version took: 0.0842 seconds

Now, the vectorized way:


python
start = time.time()

discounted_vectorized = order_totals * 0.92

vectorized_time = time.time() - start
print(f"Vectorized version took: {vectorized_time:.4f} seconds")

Output / Note

Vectorized version took: 0.0019 seconds

Your exact numbers will differ depending on your machine, but the shape of the result won't: the vectorized version will consistently come out dramatically faster — often 30 to 80 times, sometimes more, depending on the size of the data. And this is just half a million rows. Real pipelines regularly work with tens of millions.

Let's confirm both approaches actually produced the same answer, because a fast wrong answer is worse than a slow right one:


python
import numpy as np

print(np.allclose(discounted_loop, discounted_vectorized))

Output / Note

True

np.allclose() checks whether two arrays of numbers are equal, allowing for the tiny rounding differences that are completely normal in floating-point math. This is a small preview of exactly the kind of check you'll build properly into real tests, starting in Chapter 17.

You've now proven, on your own machine, with your own numbers, the entire reason this chapter exists. That instinct from Lesson 11.1 — "pause before you loop over numbers, and ask if NumPy can do it instead" — isn't a rule to memorize. It's something you just watched save you two orders of magnitude of time, yourself.

Try It Yourself: Repeat this benchmark with order_count = 5_000_000 instead of 500,000. Watch how much longer the loop version takes to run, and notice how little the vectorized version's time changes by comparison.