Python Foundation for Data Engineers

Chapter 20 — Calling APIs with Python: The requests Library

Lesson 20.1 — Your First GET Request

Time to do in Python exactly what you did by hand in your browser last chapter. Python's most widely used tool for making HTTP requests is called requests, and once installed, it turns the entire request-and-response conversation from Chapter 19 into just a couple of lines of code.

Install it:

bash
pip install requests

Now, your first real API call:

python
import requests

response = requests.get("https://jsonplaceholder.typicode.com/posts/1")

print(response.status_code)
print(response.text)
Output / Note

200 { "userId": 1, "id": 1, "title": "sunt aut facere repellat provident occaecati excepteur", "body": "quia et suscipit\nsuscipit recusandae..." }

requests.get() sends a GET request to the URL you provide, and waits for the response. response.status_code gives you exactly the number from Chapter 19.2 — 200 here means everything went fine. response.text shows you the raw response body — but notice it's just one long block of text right now, not yet a proper Python dictionary you can work with.

That's where .json() comes in:

python
data = response.json()

print(data["title"])
print(type(data))
Output / Note

sunt aut facere repellat provident occaecati excepteur <class 'dict'>

.json() parses that raw text and converts it directly into a real Python dictionary — the exact same shape you've been working with since Week 1, Chapter 4. From this point on, everything you already know about dictionaries — looking things up by key, checking with .get(), looping through them — applies directly.

Let's confirm the status code before trusting the data, a genuinely important habit worth building from your very first API call:

python
if response.status_code == 200:
    data = response.json()
    print(f"Post title: {data['title']}")
else:
    print(f"Request failed with status: {response.status_code}")
Output / Note

Post title: sunt aut facere repellat provident occaecati excepteur

This small if check is the seed of everything Chapter 22 builds properly — never assume a request worked just because you sent it. Always check the status first, exactly the way Chapter 19.2 taught you to read it.

Lesson 20.2 — Query Parameters & Headers

Real API requests are rarely as bare as requests.get(url). You'll usually want to narrow down exactly what you're asking for, using query parameters, and sometimes attach extra information about the request itself, using headers. Let's learn both properly, using a tool built specifically for seeing exactly what you send.

python
import requests

response = requests.get("https://httpbin.org/get", params={"category": "electronics", "limit": 5})
data = response.json()

print(data["url"])
print(data["args"])
Output / Note

https://httpbin.org/get?category=electronics&limit=5 {'category': 'electronics', 'limit': '5'}

httpbin.org is another free, public practice service — this one built specifically to echo back exactly what it received, which makes it genuinely excellent for learning. Notice what happened: you passed a plain Python dictionary as params, and requests automatically built the proper URL, adding a ?, joining each key and value with =, and separating multiple parameters with &. This is exactly what you saw appear in Week 1's regex lessons as URL query strings — now you know precisely how they're built, and you never have to construct one by hand.

Headers work in a very similar way — a dictionary, but describing metadata about the request itself rather than filtering the data:

python
headers = {
    "User-Agent": "python-de-foundations-course",
    "Accept": "application/json",
}

response = requests.get("https://httpbin.org/headers", headers=headers)
print(response.json())
Output / Note

{'headers': {'Accept': 'application/json', 'Host': 'httpbin.org', 'User-Agent': 'python-de-foundations-course', ...}}

User-Agent identifies what's making the request — some APIs check this and reject requests that don't identify themselves clearly, which is considered good practice regardless. Accept tells the server what format you'd like back — application/json is what you'll specify constantly in this course.

The practical difference worth remembering: query parameters change what data you get back — filters, search terms, pagination controls, all things you'll use constantly starting in Chapter 21. Headers describe how the request itself should be handled — authentication, format preferences, identification. You'll use both together, regularly, from here on.

Lesson 20.3 — API Keys & Basic Auth

Plenty of real APIs won't hand over data to just anyone — they require you to prove who you are first. The most common approach you'll meet as a data engineer is an API key: a long, unique string, issued to you, that you attach to every request as proof of identity.

There isn't one single standard for exactly how a key gets attached — it genuinely varies by API — but two patterns cover the vast majority of what you'll encounter. The first is as a header:

python
import requests

api_key = "demo-key-12345"
headers = {"Authorization": f"Bearer {api_key}"}

response = requests.get("https://httpbin.org/headers", headers=headers)
print(response.json()["headers"]["Authorization"])
Output / Note

Bearer demo-key-12345

That Bearer prefix is a genuinely common convention — it simply announces "the text that follows is an authentication token." Some APIs use this exact pattern; others use their own custom header name, like X-API-Key. Always check an API's documentation for its specific expectation — the underlying idea, attaching proof of identity to every request, stays the same regardless.

The second common pattern is Basic Authentication — a username and password, sent together, encoded automatically by requests:

python
import requests
from requests.auth import HTTPBasicAuth

response = requests.get(
    "https://httpbin.org/basic-auth/demo-user/demo-pass",
    auth=HTTPBasicAuth("demo-user", "demo-pass")
)

print(response.status_code)
print(response.json())
Output / Note

200 {'authenticated': True, 'user': 'demo-user'}

That specific httpbin.org URL is genuinely built to test exactly this — it only returns 200 if the username and password you sent match what's baked into the URL itself. Try it with the wrong password, and see what you get back:

python
response = requests.get(
    "https://httpbin.org/basic-auth/demo-user/demo-pass",
    auth=HTTPBasicAuth("demo-user", "wrong-password")
)

print(response.status_code)
Output / Note

401

There's that 401 Unauthorized from Chapter 19.2, appearing exactly where you'd expect it — wrong credentials, request rejected. One genuinely important, forward-looking note: never write a real API key or password directly into your code, the way we just did here for learning purposes. You'll learn the proper, safe way to handle real secrets in Chapter 21.1, using environment variables instead — keep that concern in mind as we move forward.

Lesson 20.4 — Hands-On: Pull Real Data from a Public API

Let's bring this chapter together by pulling a genuinely real, complete dataset from a live API, and turning it directly into something you already know how to work with from Week 2 — a DataFrame.

python
import requests
import pandas as pd

response = requests.get("https://jsonplaceholder.typicode.com/users")

if response.status_code == 200:
    users = response.json()
    print(f"Retrieved {len(users)} users")
else:
    print(f"Request failed: {response.status_code}")
Output / Note

Retrieved 10 users

Notice that status check again — the same habit from Lesson 20.1, now protecting real work. Let's look at the shape of one record before doing anything else with it, a genuinely good habit whenever you meet a new API for the first time:

python
print(users[0])
Output / Note

{'id': 1, 'name': 'Leanne Graham', 'email': 'Sincere@april.biz', 'address': {...}, 'phone': '1-770-736-8031 x56442', 'website': 'hildegard.org', 'company': {...}}

Now, your task: pull out just the fields you actually need — id, name, email, and the city, buried inside the nested address field — and build a clean DataFrame from them.

python
cleaned_users = []
for user in users:
    cleaned_users.append({
        "id": user["id"],
        "name": user["name"],
        "email": user["email"],
        "city": user["address"]["city"],
    })

df = pd.DataFrame(cleaned_users)
print(df)
Output / Note

id name email city 0 1 Leanne Graham Sincere@april.biz Gwenborough 1 2 Ervin Howell Shanna@melissa.tv Wisokyburgh 2 3 Clementine Bauch Nathan@yesenia.net McKenziehaven ...

Notice user["address"]["city"] — reaching two levels deep into the nested dictionary, exactly the way you practiced with nested dictionaries back in Week 1, Chapter 4, and again with API JSON in Chapter 19.3. You'll meet a faster way to do this flattening automatically in Chapter 23, using pd.json_normalize() — but building it by hand once, like this, makes sure you genuinely understand what that tool will be doing for you later.

You've just pulled real, live data from a real server on the internet, and turned it into a clean, usable DataFrame — the exact same DataFrame skills from Week 2, now fed by a live source instead of a file you built yourself. That's a genuinely significant milestone in this course.