Production Python for Data Engineers

Module 7: Concurrency, Async, Working at Scale

What you'll learn in this module

By the end of this module, you'll be able to:

  • Explain what concurrency actually means, and why it's different from doing things faster
  • Read and write basic async/await Python code, and explain what the event loop is actually doing when your code runs
  • Explain why unbounded concurrency can work perfectly in testing and fail badly in production, with no code changes in between
  • Bound concurrent work with a semaphore, and explain what number to actually choose and why
  • Process a dataset far larger than any single batch should handle, using a generator that never loads it all into memory at once
  • Actually profile a concurrency fix — before and after — instead of assuming it worked
  • Spot the single most common AI mistake with async code: unbounded fan-out with no limit at all

This module assumes you've never written async Python before. If you have, some of the early material will move fast — that's fine, the real content starts once the vocabulary is out of the way.

Let's begin.

Let's start with a question

Imagine you're cooking dinner, and it involves boiling pasta and making a sauce. Do you boil the pasta, wait, staring at the pot until it's done, and only then start the sauce? Or do you start the water, and while it's heating up — which takes a while, and needs none of your attention — you start chopping vegetables for the sauce?

Almost nobody actually cooks the first way. Waiting idly for water to boil, when you could be doing something else useful during that exact wait, feels obviously wasteful the moment you picture it.

Here's the thing: a huge amount of code written by data engineers does exactly the "stare at the pot" version of this, every single day — fetching from one API, waiting, then the next, waiting, then the next — when most of that time isn't the CPU doing work at all. It's just waiting. This module is about not staring at the pot.

Why this matters in data engineering

Ask yourself something concrete: when your code calls an external API and waits for a response, what is your computer actually doing during that wait?

The honest answer, almost always: nothing. The network request went out, and your program is just sitting there, doing no computation at all, until a response comes back. This is called being I/O-bound — bound by the speed of input/output (network, disk, a database), not by how fast your CPU can compute something.

Data engineering work is full of exactly this. Fetching pages from a paginated API. Querying a database. Writing to a warehouse. Calling a notification service. In almost every one of these, your program spends far more time waiting than computing.

Here's why that matters so much: if you fetch fifty things one at a time, and each fetch takes 200 milliseconds of pure waiting, you've just spent ten full seconds doing essentially nothing — one wait after another, in sequence, when nothing about those fifty requests actually depended on each other. If you could somehow start many of those waits at the same time, the total time wouldn't be the sum of all the waits — it would be close to the length of the single longest one. That difference, at real scale, is the difference between a pipeline that takes ten seconds and one that takes hundreds.

The core idea: doing many things at once, without needing more CPUs

Concurrency is not the same as parallelism

Parallelism means doing multiple things at the exact same instant — this genuinely requires multiple CPUs, or multiple cores, each doing real work simultaneously.

Concurrency means managing multiple things that are in progress at once, even on a single CPU — by intelligently switching between them whenever one of them is waiting instead of working. The chef doesn't need two pairs of hands to start water boiling and then chop vegetables. One chef, one pair of hands, but two things "in progress" at once, because most of the water's boiling time needed zero attention.

This distinction matters because it tells you why this module reaches for async instead of, say, running things on multiple threads or processes. Threads and processes are real tools, useful especially for CPU-bound work — heavy computation that genuinely needs to happen in parallel. But they come with real costs: each thread needs its own memory and scheduling overhead, and Python's own internals limit how much true CPU parallelism threads can actually achieve anyway. For I/O-bound work — mostly waiting, not computing — async gives you a much cheaper way to have thousands of things "in progress" at once, all on a single thread, because waiting, in async code, costs almost nothing.

The building blocks: async, await, and the event loop

Let's build this up from nothing, with the smallest possible example.

async def declares a function as a coroutine function. Calling it doesn't run its body immediately — it hands you back a coroutine object, like a recipe you haven't started cooking yet.

await is what actually runs a coroutine — and critically, if that coroutine is waiting on something (like a network call), await lets your program go do other useful work during that wait, instead of freezing until it's done.

The event loop is the thing coordinating all of this — it's what actually decides "this coroutine is waiting, let me go run a bit of that other one instead," switching between many in-progress coroutines as they alternate between working and waiting.

asyncio.run(...) is how you actually start the event loop and run your top-level coroutine — the entry point into all of this.

Here's the smallest possible demonstration. Two fake fetches, each taking one second — run one after another first:

python
import asyncio
import time

async def fake_fetch(name: str, seconds: float) -> str:
    print(f"Starting {name}...")
    await asyncio.sleep(seconds)
    print(f"Finished {name}")
    return name

async def main() -> None:
    start = time.perf_counter()
    await fake_fetch("A", 1)
    await fake_fetch("B", 1)
    print(f"Total: {time.perf_counter() - start:.2f}s")

asyncio.run(main())

Run this, and you'll see it take about 2 seconds — A starts and finishes completely, then B starts. Nothing concurrent happened at all; await here just ran one coroutine fully before moving to the next.

Now change exactly one thing — run them together, with asyncio.gather:

python
async def main() -> None:
    start = time.perf_counter()
    await asyncio.gather(
        fake_fetch("A", 1),
        fake_fetch("B", 1),
    )
    print(f"Total: {time.perf_counter() - start:.2f}s")

This takes about 1 second, not 2 — and if you watch the printed output, you'll see Starting A and Starting B both happen before either one finishes. Both fake fetches were "waiting" on their asyncio.sleep at the same time, on the same single thread, because neither one needed the CPU during that wait — the event loop simply let them both be in progress simultaneously.

That's the entire mechanism this module builds on. Everything below is this same idea, applied to a real, larger problem.

The real problem: unbounded fan-out

asyncio.gather(*[some_async_call(x) for x in huge_list]) schedules every single one of those calls essentially at once. For ten items, that's fine. For forty thousand, you're not making forty thousand polite requests — you're launching what looks, from the other side, indistinguishable from a burst attack.

Here's the part that makes this bug genuinely dangerous: it doesn't show up in small-scale testing. Staging environments, sample data, quick manual runs — all of these tend to use far fewer items than production actually sees. The bug is invisible below a certain scale, and guaranteed above it. That's exactly why it survives code review and testing, and only shows up for real once it's already live.

Bounding with a semaphore

An asyncio.Semaphore(n) limits how many pieces of code can be inside a specific block at the same time, to at most n. Wrap each concurrent call in async with semaphore:, and no matter how many tasks you schedule, only n of them are ever actually running at once — the rest wait their turn.

The number you choose matters. Too low, and you're needlessly slow — barely better than sequential. Too high, and you're back to overwhelming whatever you're calling. The right number is tied to the actual capacity of what you're calling, not a round number that felt reasonable.

A different problem: too much to even schedule at once

Here's something the semaphore doesn't fix. asyncio.gather(*tasks) needs the full list of tasks before it runs any of them — even bounded by a semaphore, Python still has to create every single task object up front. For fifty items, that's nothing. For a catalog of half a million, creating half a million task objects before a single one has even started running is a real memory cost, entirely separate from the concurrency problem the semaphore already solved.

The fix: chunking with a generator

The fix is to never hold the whole dataset's worth of work in memory at once. A generator can yield one manageable chunk at a time — say, fifty items — and only that chunk's tasks get created and run, before moving to the next chunk. The dataset itself can even be a generator too — reading from a file, a database cursor, a paginated API — so at no point does the full dataset exist in memory as one object.

This module's lab implements this directly: a small batch uses bounded concurrency alone, and a much larger catalog uses chunking and bounded concurrency together — two different techniques, solving two different problems, stacked on top of each other.

Manual lab: fast and wrong vs. fast and right, at real scale

Getting the lab files

Download module-7-materials.zip:

bash
cd ~/courses/pp4de
cp /mnt/c/Users/yourname/Downloads/module-7-materials.zip course-materials/
cd course-materials
unzip module-7-materials.zip

Commit it:

bash
cd ~/courses/pp4de
git add course-materials/module-7
git commit -m "Add Module 7 lab materials"
git push

The scenario

You've inherited a tool that checks current prices for a batch of SKUs against a pricing service. The service has a real limit — it can only handle 5 requests at the same time before it starts rejecting them. Your predecessor tested this tool with a handful of SKUs and it worked fine. Today, someone runs it against the full catalog — fifty SKUs.

Reproduce the problem first

  1. Go into the starter folder: 
    bash
    cd course-materials/module-7/starter
  2. Run it: 
    bash
    python3 run.py
  3. Watch the output. You should see something like Priced 5 of 50 SKUs in 0.05s, with 45 failure lines above it, all saying roughly the same thing — too many requests in flight at once.
  4. Open price_checker.py. Find asyncio.gather(*tasks, ...). Notice there's no limit anywhere on how many of those tasks actually run at the same time — all fifty get scheduled essentially at once.
  5. Open pricing_api.py and read the comment at the top. This service genuinely can't handle more than 5 requests simultaneously — that's not a made-up restriction, it's meant to represent a real, documented capacity limit, the kind every real external service has.

Notice something worth sitting with: this ran fast — 0.05 seconds. Fast isn't the problem. Only pricing 5 out of 50 SKUs, silently, is the problem.

Your task

Fix this in two stages.

Stage 1 — bound the concurrency, so that:

  1. No more than a safe number of requests are ever in flight at the same time — comfortably under the service's real limit of 5.
  2. All 50 SKUs get priced successfully.
  3. It's still meaningfully faster than pricing them one at a time.

Stage 2 — handle a much bigger catalog. Imagine the real catalog isn't 50 SKUs, but 500 (or more). Write a generator that yields SKUs in chunks, and process the catalog chunk by chunk, using your bounded check_all_prices within each chunk. Prove — don't just assume — that this never holds the whole catalog's tasks in memory at once.

A question worth asking before you look at the solution

Same habit as every module. If the solution looks different, ask why.

Here's the honest answer: the core idea — fetch a price for a SKU — is identical throughout. What's added is entirely about pacing, in two separate layers: a semaphore controlling how many fetches run at once, and a generator controlling how many fetches even get created at once. Neither one changes what a single fetch does.

Full worked solution

The complete solution lives in course-materials/module-7/solution/:

solution/
├── pyproject.toml
├── run.py
└── src/
    └── pricecheck/
        ├── __init__.py
        ├── pricing_api.py      # unchanged - the simulated external service
        ├── price_checker.py    # stage 1: a semaphore bounding concurrency
        └── catalog.py           # stage 2: chunked, generator-based processing

price_checker.py — stage 1, the semaphore:

python
import asyncio

from pricecheck.pricing_api import fetch_price

MAX_CONCURRENT_REQUESTS = 4


async def _fetch_one(semaphore: asyncio.Semaphore, sku: str) -> tuple[str, float | Exception]:
    async with semaphore:
        try:
            price = await fetch_price(sku)
            return sku, price
        except Exception as e:
            return sku, e


async def check_all_prices(skus: list[str]) -> dict[str, float]:
    semaphore = asyncio.Semaphore(MAX_CONCURRENT_REQUESTS)
    tasks = [_fetch_one(semaphore, sku) for sku in skus]
    results = await asyncio.gather(*tasks)

    prices: dict[str, float] = {}
    for sku, result in results:
        if isinstance(result, Exception):
            print(f"Failed to price {sku}: {result}")
        else:
            prices[sku] = result
    return prices

Notice MAX_CONCURRENT_REQUESTS = 4 — deliberately below the service's actual limit of 5, leaving a small safety margin.

catalog.py — stage 2, the new piece this module adds:

python
from collections.abc import Iterable, Iterator

from pricecheck.price_checker import check_all_prices

CHUNK_SIZE = 50


def iter_sku_chunks(skus: Iterable[str], chunk_size: int = CHUNK_SIZE) -> Iterator[list[str]]:
    """Yields successive chunks of `chunk_size` SKUs, one at a time.
    `skus` can itself be a generator - this never materializes the
    whole catalog into memory, whether it's 500 SKUs or 500,000.
    """
    chunk: list[str] = []
    for sku in skus:
        chunk.append(sku)
        if len(chunk) == chunk_size:
            yield chunk
            chunk = []
    if chunk:
        yield chunk


async def check_full_catalog(
    skus: Iterable[str], chunk_size: int = CHUNK_SIZE
) -> dict[str, float]:
    """Processes an entire catalog, chunk by chunk. Within each chunk,
    check_all_prices still applies its own bounded concurrency - the
    two techniques stack, they don't replace each other.
    """
    all_prices: dict[str, float] = {}
    for chunk_number, chunk in enumerate(iter_sku_chunks(skus, chunk_size), start=1):
        print(f"Processing chunk {chunk_number} ({len(chunk)} SKUs)...")
        chunk_prices = await check_all_prices(chunk)
        all_prices.update(chunk_prices)
    return all_prices

Notice iter_sku_chunks is a generator function — it uses yield, not return. That's what makes it lazy: it only produces the next chunk when something actually asks for it, so the full catalog is never sitting in memory as one object, no matter how large it is.

Verify it, step by step — with real proof, not assumptions

  1. Create a virtual environment and install: 
    bash
    cd course-materials/module-7/solution
python3 -m venv .venv
source .venv/bin/activate
pip install -e .
pip install mypy
  2. Run mypy --strict: 
    bash
    python -m mypy --strict src/pricecheck/ run.py
    Expected: Success: no issues found.
  3. Run it: 
    bash
    python run.py
    You should see Priced 50 of 50 SKUs for the first batch, then ten chunk-progress lines, ending in Priced 500 of 500 catalog SKUs.
  4. Now prove the chunking is genuinely lazy — not just "happens to work." Run this: 
    bash
    python3 -c "
from pricecheck.catalog import iter_sku_chunks

generated_count = 0

def tracked_catalog(n):
    global generated_count
    for i in range(n):
        generated_count += 1
        yield f'SKU-{i:05d}'

chunks = iter_sku_chunks(tracked_catalog(1000), chunk_size=50)
next(chunks)
print(f'After ONE chunk: {generated_count} generated (should be 50, not 1000)')
next(chunks)
print(f'After TWO chunks: {generated_count} generated (should be 100, not 1000)')
"
    If this shows 50 and then 100 — not 1000 both times — you've proven the generator never materializes the whole catalog, exactly the claim this module makes.

If step 3 succeeded and step 4 showed genuinely lazy generation, you've verified both halves of what this module set out to teach.

AI-assisted round

The task

Ask your assistant to extend the solution:

Output / Note

"Add a function that also fetches each SKU's current stock level from a second, similar service, check_stock_levels, for the same list of SKUs, at the same time as pricing. Write a test for it, and run it before telling me you're done."

The known failure pattern to watch for

This is the exact bug from this module's opening story: an AI assistant reaching for asyncio.gather on a new fan-out, without reusing the concurrency bound you already established.

Here's what that tends to look like:

diff
+ async def check_stock_levels(skus: list[str]) -> dict[str, int]:
+     tasks = [fetch_stock_level(sku) for sku in skus]
+     results = await asyncio.gather(*tasks)
+     return dict(zip(skus, results))

This runs fine in a quick test with a handful of SKUs — the same reason the original bug survived staging. At real scale, against a real service with a real capacity limit, this reintroduces exactly the problem this module already fixed, in a second function sitting right next to the first one that correctly uses a semaphore.

The guardrail

If you catch this, write it down:

Output / Note

Module 7 guardrail: Any time an AI assistant adds a new asyncio.gather call over a list of items, check specifically whether it's bounded. If a semaphore pattern already exists elsewhere in the project, check whether the new code reuses it — a second, unbounded fan-out function is easy to miss in review because it looks like ordinary async code, and it will pass any test that doesn't exercise it at real scale.

If your assistant reused the existing bounded pattern without being told to — log that too.

Common mistakes

  • Forgetting await. Calling an async def function without await doesn't run it — it just gives you back an unused coroutine object, silently. Python will usually warn about this ("coroutine was never awaited"), and it's worth taking that warning seriously the first time you see it, not dismissing it.
  • Calling a blocking function inside an async def. A genuinely slow, non-async call (like time.sleep instead of asyncio.sleep, or a synchronous database driver) freezes the entire event loop while it runs — not just the coroutine that called it. This defeats the entire point of using async in the first place.
  • asyncio.gather with no limit at all. Works perfectly at small scale. Guaranteed to cause real problems once the input size grows enough — and "enough" is often exactly what production looks like and testing doesn't.
  • Setting the concurrency limit right at the documented capacity, with no margin. Leaves zero room for the real world being slightly less generous than the documentation.
  • Loading an entire large dataset into memory before processing any of it, when a generator would let you start immediately and use a fraction of the memory — and not proving it's actually lazy, just assuming a generator automatically means "memory-safe."

Capstone tie-in

Step 1: Open your capstone repo and start the infrastructure

Confirm the title bar says pp4de [WSL: Ubuntu].

bash
docker compose start
docker compose ps

Step 2: Bound your capstone's concurrent page fetches

In VS Code's Explorer:

  1. Right-click pipeline (inside src/) → New File → name it fetcher.py, if you don't already have one from earlier modules.
  2. Write an async function that fetches multiple pages from the mock API concurrently, bounded by a semaphore — the same pattern as this module's lab, applied to real pages of records instead of SKUs. Remember Module 3's retry logic and Module 6's structured logging — this is a good place to use both together.

Step 3: Add a chunked, generator-based transform

Wherever your capstone processes a batch of validated records (from Module 2's models), rewrite it — or write it fresh — as a generator that yields transformed records one at a time, the same shape as this module's iter_sku_chunks. Prove it's genuinely lazy the same way the lab did — track how many records get pulled through before the first chunk is consumed, and confirm it's not the whole dataset.

Step 4: Profile it for real — the one mandatory terminal step

Open the integrated terminal, confirm your venv is active, and measure your bounded fetch against a real, larger date range from the mock API — time it, the same way this module's lab did, and write down the actual number. Compare it against what a naive sequential fetch of the same pages would cost, using time.perf_counter() around both.

Step 5: Commit and push through VS Code's Source Control panel

  1. Click the Source Control icon in the sidebar.
  2. Confirm .venv is not listed under Changes.
  3. Stage, commit with a message like Module 7: bounded concurrent fetch, chunked transform, profiled, and click Commit.
  4. Click Sync Changes to push to GitHub.
  5. Confirm the changes appear on GitHub.

Check row 7 off your SPEC.md checklist. Two to go.

Before you close for the day:

bash
docker compose stop

Interview drill

Every question below follows the same pattern. First, the question. Then, what the interviewer wants to hear. Then, a junior engineer's answer. Then, a strong senior answer.

Recall

Question: "What's the actual difference between concurrency and parallelism? And why does Python's asyncio help with I/O-bound work specifically, rather than CPU-bound work?"

What the interviewer wants to hear: a precise distinction between the two terms, and a real explanation of why async specifically suits waiting-heavy work, not just "it's faster."

Junior answer: "Concurrency and parallelism are basically the same thing — doing multiple things at once. Async makes code run faster."

(Conflates two genuinely different concepts, and "makes code run faster" isn't accurate or specific — async doesn't speed up computation at all.)

Senior answer: "Parallelism is doing multiple things at the exact same instant, which needs multiple CPUs actually working simultaneously. Concurrency is managing multiple things that are in progress at once, by switching between them — which doesn't require more than one CPU, as long as most of the 'in progress' time is spent waiting, not computing. asyncio specifically helps with I/O-bound work — network calls, database queries — because during those waits, the CPU has nothing to do anyway, so switching to another task costs almost nothing. It wouldn't help nearly as much with CPU-bound work, like heavy number crunching, because there's no idle waiting time to fill with other tasks in the first place — that's a genuinely different problem, better suited to multiprocessing."

Debugging

Question: shown as a real snippet and incident story, with no explanation yet:

python
import asyncio
import aiohttp

async def fetch_player_stats(session: aiohttp.ClientSession, player_id: str) -> dict:
    async with session.get(f"https://stats.internal/players/{player_id}") as resp:
        return await resp.json()

async def refresh_leaderboard(player_ids: list[str]) -> list[dict]:
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_player_stats(session, pid) for pid in player_ids]
        results = await asyncio.gather(*tasks)
    return results

"This runs fine in staging, where tournaments have a few hundred players. In production, a tournament with 40,000 players caused this function to start throwing connection errors partway through, and separately, the downstream stats.internal service had its own on-call paged for what looked like a traffic spike attack. No code changed between staging and prod. What's actually going on, and why did staging never catch it?"

What the interviewer wants to hear: identifying unbounded concurrency as the root cause — not vague guesses like "the network is unreliable" — and explaining precisely why staging's smaller scale hid the bug.

Junior answer: "Maybe aiohttp isn't handling that many requests well, or the network had some issue at that scale."

(Guesses at plausible-sounding external causes instead of examining what the code itself actually does at scale.)

Senior answer: "asyncio.gather schedules every single task here essentially at once — for 40,000 players, that's 40,000 requests fired close to simultaneously, with nothing bounding how many are actually in flight. That exhausts the calling client's own connection pool, which explains the connection errors on this side. From stats.internal's point of view, a burst of 40,000 near-simultaneous requests looks indistinguishable from a traffic spike attack, which explains their on-call getting paged separately. Staging never caught this because a few hundred players never exposed the missing bound — this bug scales with input size, so it's invisible below some threshold and guaranteed above it, and a functional test with realistic-looking but small data will never catch that. The fix is bounding concurrency with a semaphore, sized to what stats.internal can actually handle, not an arbitrary round number."

AI-review

Question: shown this diff, with no explanation yet:

diff
+ async def check_stock_levels(skus: list[str]) -> dict[str, int]:
+     tasks = [fetch_stock_level(sku) for sku in skus]
+     results = await asyncio.gather(*tasks)
+     return dict(zip(skus, results))

The project already has an established, working pattern for bounded concurrency — a semaphore, used consistently in check_all_prices for a very similar kind of fan-out.

What the interviewer wants to hear: recognizing that this new function doesn't reuse the existing bounded pattern, and explaining why that's a real risk, not just a style inconsistency.

Junior answer: "It's a bit inconsistent with the other function's style, but functionally it should work fine for fetching stock levels."

(Notices the inconsistency but treats it as cosmetic, not a real correctness or reliability risk.)

Senior answer: "This is the exact same unbounded fan-out problem the rest of this project already solved, reintroduced in a new function. asyncio.gather(*tasks) here has no semaphore at all — every SKU's stock level gets fetched essentially simultaneously, with nothing bounding it. It'll probably work fine in a quick test with a handful of SKUs, which is exactly why this kind of bug survives review — it looks identical to correct async code until it runs at real scale. I'd tell the assistant to reuse the same bounded pattern as check_all_prices — wrap each call with the existing semaphore, or a new one sized to whatever service fetch_stock_level actually calls."

Judgment

Interviewer: "You're the on-call engineer for a logistics platform that ingests scan events from warehouses and delivery vehicles, to update the live status of packages in transit. During a peak shipping day, the ingestion service starts falling behind — events are queuing up faster than they're being processed. Your team lead proposes removing the concurrency limit on the downstream status-update calls entirely, to 'let it process as fast as it possibly can.' What do you say?"

Candidate (asking first): "Before I answer — do we know what's actually rate-limiting throughput right now? Is the downstream status-update service itself the bottleneck, or is something upstream of it, like the event queue itself, actually the constraint?"

Interviewer: "Good question — we've confirmed the downstream status-update service is the bottleneck. It has a documented capacity, and our current concurrency limit is already set right at that documented number."

Candidate (first answer): "Then I'd push back on removing the limit entirely — that's very likely to make things worse, not better. If we're already at the service's documented capacity, removing the limit doesn't unlock more real throughput, since the service still physically can't process more than it can process. What it will do is start generating errors and retries once we exceed that real capacity, which could easily make our effective throughput worse than what we have now, not better."

Interviewer (pushback): "But we're falling behind right now — isn't doing something drastic better than watching the queue keep growing?"

Candidate (round 1): "I understand the urgency, but 'drastic' and 'effective' aren't the same thing here. If the downstream service is genuinely the bottleneck, unbounding our concurrency doesn't increase its actual capacity — it just means we start hitting that capacity limit with errors and retries instead of hitting it cleanly. That usually makes backlogs worse, not better."

Interviewer (second pushback): "So what would you actually propose to relieve the backlog today?"

Candidate (round 2): "A few things, in order of how fast they could help. First, confirm our current concurrency limit is actually optimally tuned — right at documented capacity might not be the same as actual current real-world capacity, especially under today's unusual peak load; there might be a little real headroom to find carefully, by testing a modest, deliberate increase and watching error rates closely. Second, I'd look at whether we can reduce the number of status-update calls needed in the first place — batching multiple scan events per package into a single downstream update, if the API supports it, instead of one call per individual event."

Interviewer (final challenge): "Your team lead still thinks removing the limit is the fastest fix. Convince them otherwise, directly."

Candidate (final defense): "I'd frame it around what 'fast' actually means here. Removing the limit feels fast because it's a one-line change, but if the downstream service is genuinely at capacity, the real-world result is more errors, more retries, and likely a worse effective throughput than we have right now. A careful, measured increase to the limit, backed by watching real error rates, or reducing the number of calls we need to make at all through batching, both directly address the actual bottleneck. Those take a little more thought than deleting a limit, but they're actually likely to help, instead of turning a backlog into a backlog plus a wave of failed requests."

Model strong answer — a single answer, given upfront:

Output / Note

"I'd want to confirm first whether the downstream status-update service is actually the bottleneck, or whether something upstream is — assuming it's confirmed the downstream service is at its documented capacity, I'd push back strongly on removing the concurrency limit entirely. If that service genuinely can't process more than it can process, removing the limit doesn't create more real capacity — it just means we start hitting that same limit with errors and retries instead of hitting it cleanly, which typically makes effective throughput worse, not better, during exactly the peak load we're trying to survive. I'd propose two things instead: cautiously testing whether the documented capacity has a little real headroom under today's specific conditions, adjusted carefully and watched closely for error rates — and separately, looking at whether scan events can be batched into fewer downstream calls per package, which reduces actual load on the bottleneck rather than just changing how hard we push against it."