Chapter 21 — Secrets, Pagination & Rate Limits

Lesson 21.1 — Keeping Secrets Out of Your Code: Environment Variables & .env

Back in Chapter 20.3, you saw an API key written directly into a line of Python. That was fine for learning, but it's a genuinely serious habit to break before you write real, production code — because that code, with the key sitting right inside it, tends to end up committed to Git, shared with teammates, or pasted into a chat message, all without anyone meaning to leak a secret.

The fix is separating your secrets from your code entirely, using environment variables — values that live outside your script, in the environment your script runs in, rather than inside the file itself.

Let's set one up. First, install a small, genuinely standard helper library:


bash
pip install python-dotenv

Now, in your python-de-foundations folder, create a new file called .env — yes, starting with a dot, no other name before it:

API_KEY=demo-key-12345

That's it — just the name of the variable, an equals sign, and the value, no quotes needed. Now, in your Python script, load it:


python
from dotenv import load_dotenv
import os

load_dotenv()

api_key = os.getenv("API_KEY")
print(f"Loaded key starting with: {api_key[:4]}...")

Output / Note

Loaded key starting with: demo...

load_dotenv() reads your .env file and makes every value inside it available through os.getenv() — the same os module you first met back in Week 2, Chapter 16, checking file sizes. Notice we only printed the first four characters here, not the whole key — a small, genuinely good habit, since printing a real secret anywhere, even to your own terminal during testing, is a way secrets end up somewhere they shouldn't.

Now, the single most important step, easy to forget and genuinely important not to: tell Git to never track this file. Create a file called .gitignore in the same folder, if you don't have one already, and add this line:

.env

This means your .env file — and the real secret inside it — stays on your machine only, and never gets uploaded to GitHub or shared alongside your code. Your actual script, meanwhile, contains no secrets at all — just a line asking the environment for one by name, which is exactly what makes it safe to share, commit, and hand to a teammate.

The practical habit to carry forward, for the rest of this week and beyond: never type a real API key, password, or token directly into a .py file. Always load it from an environment variable, and always make sure the file holding the real value is excluded from version control.

Lesson 21.2 — Pagination: Getting All the Data, Not Just Page One

Real APIs almost never hand you an entire dataset in a single response. Ask for "all orders," and a well-behaved API will hand you back the first hundred, along with a way to ask for the next hundred — a system called pagination. Missing this, and only ever reading the first page, is a genuinely common, quiet bug: your pipeline runs successfully, reports no errors, and simply never notices it only processed a fraction of the real data.

Let's see it in action, using the practice API from this week:


python
import requests

response = requests.get(
    "https://jsonplaceholder.typicode.com/comments",
    params={"_page": 1, "_limit": 10}
)

comments = response.json()
print(f"Got {len(comments)} comments")
print(comments[0]["email"])

Output / Note

Got 10 comments Eliseo@gardner.biz

That _page and _limit pattern is genuinely common across many real APIs, though the exact parameter names vary — some use page and per_page, others use offset and limit. Always check an API's documentation for its specific pagination style; the underlying idea stays the same regardless of the exact names.

Here's the real question: how do you know when you've reached the last page, and it's time to stop asking for more? The most reliable way, and the one that works here, is simple: keep asking for the next page until a response comes back empty.


python
import requests

all_comments = []
page = 1

while True:
    response = requests.get(
        "https://jsonplaceholder.typicode.com/comments",
        params={"_page": page, "_limit": 20}
    )
    comments = response.json()

    if len(comments) == 0:
        break

    all_comments.extend(comments)
    page += 1

print(f"Total comments collected: {len(all_comments)}")

Output / Note

Total comments collected: 500

Read this like a sentence: "keep asking for pages, adding each one's results to our growing list, until a page comes back with nothing in it — then stop." That while True is the same unbounded loop idea from Week 1, Chapter 5.3, with a proper break condition inside it, exactly the pattern that lesson warned you to always include.

Notice .extend(), appearing here for the first time — it's genuinely similar to .append() from Week 1, Chapter 4, but instead of adding one single item to the end of a list, it adds every item from another list, one at a time. all_comments.append(comments) would have added the entire page as one nested list; .extend() correctly flattens it into your running collection.

Real-world caution worth carrying forward: an API that never returns an empty page — because of a bug on your end, or an API that genuinely never runs out of new data — would make this loop run forever, exactly the warning from Week 1, Chapter 5.3. In real production code, it's wise to add a safety limit, like stopping after a very high maximum page count, just in case. You'll see this practiced properly in this chapter's hands-on exercise.

Lesson 21.3 — Rate Limits: Being a Good Citizen

You met 429 Too Many Requests back in Chapter 19.2. This lesson is about what that status code actually means for you in practice, and how to write code that respects it, rather than fighting it.

Most real APIs limit how many requests you're allowed to make in a given stretch of time — say, 60 requests per minute. This isn't the API being difficult on purpose; it's protecting the server from being overwhelmed, by anyone, including well-meaning code like yours that might otherwise fire off thousands of requests in a tight loop without pausing.

Let's see a 429 response on purpose, using a service built to simulate exactly this:


python
import requests

response = requests.get("https://httpbin.org/status/429")
print(response.status_code)

Output / Note

429

In real, live use, hitting a 429 means "slow down." The simplest, most respectful response is to pause before trying again:


python
import time
import requests

urls_to_fetch = [
    "https://jsonplaceholder.typicode.com/posts/1",
    "https://jsonplaceholder.typicode.com/posts/2",
    "https://jsonplaceholder.typicode.com/posts/3",
]

for url in urls_to_fetch:
    response = requests.get(url)
    print(f"{url}: {response.status_code}")
    time.sleep(1)

Output / Note

https://jsonplaceholder.typicode.com/posts/1: 200 https://jsonplaceholder.typicode.com/posts/2: 200 https://jsonplaceholder.typicode.com/posts/3: 200

That time.sleep(1) — pausing for one second between each request — is a small, genuinely important habit: deliberately spacing out your requests, rather than firing them as fast as your code physically can, so you never come close to overwhelming a server, or triggering a rate limit in the first place.

Some APIs are even more considerate, and tell you exactly how long to wait, using a Retry-After header on a 429 response:


python
response = requests.get("https://httpbin.org/status/429")

retry_after = response.headers.get("Retry-After")
if retry_after:
    print(f"Server asked us to wait {retry_after} seconds")
else:
    print("No Retry-After header provided — falling back to a default wait")

Output / Note

No Retry-After header provided — falling back to a default wait

This particular practice endpoint doesn't include that header, but plenty of real APIs do — and the honest, respectful approach worth carrying forward is: check for a Retry-After header first, and use it if it's there; fall back to a sensible default pause, like a few seconds, if it isn't. You'll build exactly this kind of thoughtful, layered decision-making into a proper retry function in Chapter 22.

Lesson 21.4 — Hands-On: Pull a Full Paginated Dataset

Let's bring this chapter together into one properly built, safe pagination function — combining pagination, a sensible safety limit, and a respectful pause between requests, all in one place.


python
import requests
import time

def fetch_all_pages(base_url, page_size=20, max_pages=50):
    """
    Fetch every page of results from a paginated API endpoint.
    Stops when a page comes back empty, or when max_pages is hit
    as a safety limit against an endpoint that never runs out.
    Returns a single combined list of all records found.
    """
    all_records = []
    page = 1

    while page <= max_pages:
        response = requests.get(base_url, params={"_page": page, "_limit": page_size})

        if response.status_code != 200:
            print(f"Stopping — page {page} returned status {response.status_code}")
            break

        records = response.json()

        if len(records) == 0:
            print(f"Reached the end at page {page}")
            break

        all_records.extend(records)
        page += 1
        time.sleep(0.2)

    return all_records

Notice this function combines several habits from this chapter, deliberately, in one place: the empty-page stopping condition from Lesson 21.2, the max_pages safety limit warned about in that same lesson, a status code check before trusting each page, and a small, respectful pause between requests from Lesson 21.3.

Let's run it on a real, complete dataset:


python
comments = fetch_all_pages("https://jsonplaceholder.typicode.com/comments", page_size=25)
print(f"Total records collected: {len(comments)}")
print(comments[0])

Output / Note

Reached the end at page 21 Total records collected: 500 {'postId': 1, 'id': 1, 'name': '...', 'email': 'Eliseo@gardner.biz', 'body': '...'}

That's a full, real dataset of 500 comments, pulled safely and respectfully across 21 separate requests, without you having to think about any of the pagination mechanics by hand — the function handles it, the same way clean_customer_file from Week 1, Chapter 10 handled its cleaning logic behind one clean interface.

You now have a genuinely reusable tool: hand fetch_all_pages any similarly-paginated API URL, and it will safely bring back everything, page by page, without you writing the loop again from scratch.