Python Foundation for Data Engineers

Chapter 19 — How the Web Talks: HTTP Fundamentals

Lesson 19.1 — Requests and Responses: The Web's Basic Conversation

Every file you've worked with so far in this course has sat still, waiting for you — a CSV on your disk, a JSON file you created yourself. Starting this week, your data is going to come from somewhere alive: a server, somewhere else in the world, that you ask for data, and that answers you back, right now, in real time. That's what an API is — a way for your code to have a conversation with someone else's system.

Before writing a single line of Python this week, it's worth understanding what's actually happening in that conversation, because every tool you use from here on is just automating this same basic exchange.

Here's the whole idea, in plain language: your code sends a request — a message saying "I'd like this piece of information, please." A server, somewhere else, receives that request, decides how to respond, and sends back a response — the data you asked for, or an explanation of why it can't give it to you.

This is genuinely the same shape as walking up to a library desk and asking for a book. You make a specific request — "do you have this title?" The librarian responds — either handing you the book, or telling you it's not available, or that you asked incorrectly. Every API call you'll ever make follows this same request-then-response pattern, no exceptions.

A request has a few important pieces, worth knowing by name now, because you'll meet each one properly over the next few lessons:

  • A URL — the address of exactly what you're asking for, like https://api.example.com/orders
  • A method — what kind of action you're asking for. GET means "give me data." POST means "here's some new data, please save it." There are others, but as a data engineer pulling data out of systems, you'll use GET the vast majority of the time.
  • Sometimes, headers and parameters — extra details attached to the request, like an API key proving who you are, or a filter narrowing down exactly what you want back.

And a response has its own important pieces:

  • A status code — a short number telling you, immediately, whether the request succeeded, failed, or something in between. You'll learn to read these fluently in the next lesson.
  • A body — the actual content of the response, which for nearly every API you'll work with in data engineering, is JSON — the exact format you already know well from Week 1, Chapter 8 and Week 2, Chapter 16.

Keep this whole picture in mind as we go: request out, response back, and everything this week teaches you is really just about doing that conversation carefully, honestly, and resiliently — the same values that have run through this entire course, now applied to a data source that's alive and can occasionally misbehave in ways a local file never does.

Lesson 19.2 — Status Codes: Reading the Web's Traffic Lights

Every single response you get back from an API comes with a status code — a three-digit number that tells you, before you even look at the actual content, whether things went well. Learning to read these on sight is exactly like learning to read a Python traceback back in Week 1, Chapter 7 — a skill that turns a confusing failure into a precise, specific piece of information.

Status codes are grouped by their first digit, and that grouping is worth memorizing now, because it tells you the general story instantly:

2xx — Success. Your request worked. By far the most common one you'll see is 200 OK — everything went fine, here's your data. Occasionally you'll see 201 Created, meaning something new was successfully saved — less relevant for the GET requests you'll mostly be making, but worth recognizing.

4xx — You made a mistake. Something about your request was wrong. 404 Not Found means the URL you asked for doesn't exist — a genuinely common typo-driven bug. 401 Unauthorized means you didn't prove who you are, usually a missing or invalid API key. 429 Too Many Requests means you asked too often, too fast — you'll learn to handle this properly in Chapter 21's rate limits lesson.

5xx — The server made a mistake. Something broke on their end, not yours. 500 Internal Server Error is the generic version — something went wrong on the server, and there's often nothing more specific to learn from it. 503 Service Unavailable usually means the server is temporarily overloaded or down for maintenance.

Here's the genuinely important, practical distinction for you as a data engineer: 4xx errors usually mean "fix your request" — retrying the exact same request won't help. 5xx errors often mean "try again shortly" — the problem is temporary, and a retry might genuinely succeed. This distinction is going to matter directly in Chapter 22, when you build a proper retry system — a good one treats these two categories very differently, and you now know why.

You'll get hands-on practice reading real status codes very soon — for now, the goal is simply recognizing that a status code is the very first thing you check on any response, before you even look at the data inside it. A 200 with broken data inside is a very different problem than a 404 telling you the URL itself is wrong, and confusing the two wastes real debugging time.

Lesson 19.3 — JSON Payloads: What APIs Actually Send Back

Here's genuinely good news: you already know the format almost every API you'll work with actually speaks. It's JSON — the exact format from Week 1, Chapter 8, and Week 2, Chapter 16's pd.json_normalize() lesson. Nothing new to learn about the format itself here — just where it's coming from now.

A typical API response body looks exactly like the JSON files you've already built by hand:

json
{
  "id": 101,
  "name": "Priya Shah",
  "email": "priya.shah@example.com",
  "address": {
    "city": "Mumbai",
    "zipcode": "400001"
  }
}

Recognize that shape? A dictionary, with a nested dictionary inside it for address — precisely the nested structure pd.json_normalize() was built to flatten, back in Week 2. Real APIs nest data constantly — a user with an address, an order with line items, a customer with a list of past purchases — and every tool you already know for handling nested JSON transfers directly to handling API responses.

The one genuinely new detail worth knowing now, before you make your first real API call next chapter: many APIs return not just one record, but a list of them, for a single request:

json
[
  {"id": 101, "name": "Priya Shah"},
  {"id": 102, "name": "Raj Kumar"}
]

That's the exact "list of dictionaries" shape from Week 1, Chapter 4, and Week 2, Chapter 12 — a batch of records, ready to become a DataFrame in a single line, exactly the way you've already practiced.

So here's the genuinely reassuring truth to carry into this whole week: the destination format hasn't changed at all. What's changing is only how that JSON arrives — over a live network connection instead of from a file already sitting on your disk — and everything that comes with that: the request needed to fetch it, the possibility of it failing partway, and the need to handle that failure gracefully. That's what the rest of this week actually teaches.

Lesson 19.4 — Hands-On: Explore a Real Public API in the Browser

Before writing any Python this week, let's see a real request and response with your own eyes, using nothing but your web browser — because a browser, at its core, is just a tool for making GET requests and displaying whatever comes back.

Open a new tab in your browser, and go to this address:

https://jsonplaceholder.typicode.com/posts/1

This is a free, public, practice API — built specifically for exactly this kind of learning, with no signup and no API key required. You'll use it throughout this entire week, so it's worth getting comfortable with it now.

You should see something like this appear in your browser:

Output / Note

{ "userId": 1, "id": 1, "title": "sunt aut facere repellat provident occaecati excepteur", "body": "quia et suscipit\nsuscipit recusandae..." }

That's it — that's a real API response, JSON, exactly as described in the last lesson. Your browser just made a GET request to that URL, and the server responded with this single "post" record.

Now change the URL slightly, and visit:

https://jsonplaceholder.typicode.com/posts

This time, you'll see a much longer response — a full list of a hundred posts, the "list of dictionaries" shape from the last lesson, all sent back in a single response.

Now, let's deliberately trigger an error, so you can see one in its natural habitat. Visit:

https://jsonplaceholder.typicode.com/posts/99999
Output / Note

{}

An empty response — this particular practice API returns an empty object rather than a proper 404 for a missing post, which is itself a useful, honest lesson: not every API behaves identically or "correctly" — part of working with real APIs is discovering their specific quirks, something you'll only learn by actually looking closely, exactly as you're doing right now.

One more useful stop. Visit:

https://jsonplaceholder.typicode.com/users
Output / Note

[ { "id": 1, "name": "Leanne Graham", "email": "Sincere@april.biz", "address": { "street": "Kulas Light", "city": "Gwenborough", "zipcode": "92998-3874" }, "company": { "name": "Romaguera-Crona" } }, ... ]

Notice the nested address and company fields — this is the exact /users endpoint you'll come back to properly in Chapter 23, when you practice flattening nested API data into a clean DataFrame with pd.json_normalize().

You've now seen, first-hand, everything Lessons 18.1 through 18.3 described — a request, a response, a status, a JSON body, a list of records, and even a small real-world quirk. Every Python tool you learn from here on is simply about automating exactly what you just did by hand, at scale, reliably, and safely.