Your concurrency limit is not a queue limit

Ten jobs are running. Another 9,990 are waiting. The dashboard says concurrency is ten, which is correct and considerably less comforting than it sounds.

This is an easy system to build. Give each document, URL, or agent subtask its own asynchronous task. Put a semaphore around the expensive call. Everything looks civilized because the model endpoint only sees ten requests at once. Meanwhile, the process has already accepted responsibility for the other 9,990.

Those numbers come from the small, deliberately stalled program below. It makes no model calls. The mistake it demonstrates sits in the code that launches work, before intelligence has much opportunity to help.

The queue is made of tasks

Here is the shape of the problem, with process standing in for the work:

limit = asyncio.Semaphore(10)

async def run(item):
    async with limit:
        return await process(item)

await asyncio.gather(*(run(item) for item in items))

The semaphore does its job. It limits how many callers hold a permit. But asyncio.gather schedules the coroutine arguments as tasks. Creating all those wrappers happens before the permits can slow the producer down. The pending tasks become a queue, whether or not a variable named queue appears anywhere in the program.

The same pattern appears in a library. At this pinned LangChain revision, gather_with_concurrency wraps coroutines in a semaphore and passes the wrappers to asyncio.gather. Its name describes the limit it provides. That helper is not promising bounded admission, and inspecting it does not establish how every LangChain application submits work. It does show why reading a max_concurrency setting is insufficient to understand the surrounding system.

A waiting task still exists. It can retain arguments and whatever those arguments reference. If the caller constructed a large document context before scheduling it, the context waits too. Moving the model call behind a permit does not move that allocation behind the permit.

For a known, small batch, this may be entirely fine. The problem starts when a local limit gets mistaken for a limit on everything the application has accepted.

Count what has already been pulled

The complete runnable example compares two ways to process 10,000 integer jobs. One creates a task for every job and puts a ten-permit semaphore inside each task. The other uses ten workers, a queue that holds twenty items, and one producer reading a lazy source.

An event holds the workers still while the program counts the work. This makes the comparison inspectable without guessing how long to sleep or calling a model. On Python 3.14.7, it prints:

eager: active=10, waiting=9990, unfinished_tasks=10000
bounded: active=10, queued=20, held_by_producer=1, pulled=31
Both completed 10000 jobs. All count assertions passed.

The second line is the interesting one. Thirty jobs have reached a worker or the queue. The producer has pulled one more item and is waiting to place it. It has not pulled the remaining 9,969. Once the event releases the workers, both designs finish all 10,000 jobs.

The example measures counts, not memory savings, throughput, agent quality, or API cost. It exercises normal completion; a service still needs a policy for rejected, expired, and cancelled jobs. Integers make cheap jobs. Real payloads will not all cost the same. This run shows where unfinished work resides in these two implementations and whether the producer keeps pulling while the workers are stalled.

In a fixed-worker design, the producer's core loop is plain. Worker startup and shutdown are in the complete example:

queue = asyncio.Queue(maxsize=20)

for item in source:
    await queue.put(item)

Python's queue contract says put waits when a positive maxsize has been reached. Here, the producer itself waits. It cannot advance the loop to fetch the next item until that put succeeds.

Now use asyncio.create_task to schedule each queue.put(item) instead of awaiting it in the loop. You have bounded the queue's contents and created a separate population of tasks waiting to enter it. The queue can proudly report twenty while the work waits outside. The location of the await matters more than the presence of the queue.

This assumes the source is actually lazy. An upstream function that first loads every document into a list has already made a different decision. A source with its own prefetch buffer adds another place to count. Backpressure has to reach the producer that can stop acquiring work.

We had names for this

Matt Welsh, David Culler, and Eric Brewer described explicit stage queues and overload policies in SEDA in 2001. A finite queue could reject an entry, letting the application block upstream, shed load, return an error, or offer reduced service. The queue made a resource decision visible.

Reactive Streams later made demand part of the exchange between asynchronous components. The receiving side should not be forced to buffer an arbitrary amount of data merely because the sending side can produce it. Neither idea needs to be rediscovered as an agent architecture.

Java's ThreadPoolExecutor documentation is refreshingly explicit about the choice. A fixed pool can sit in front of an unbounded queue. A bounded queue with finite thread limits instead reaches a saturation point where a rejection policy applies. The worker count and the waiting-room size are separate decisions.

An agent supervisor adds a particularly enthusiastic source of work. In its June 2025 account of building a research system, Anthropic described early agents spawning fifty subagents for simple queries. The same report describes benefits from parallel research and prompt rules for allocating effort. It is an account of one system, not a claim that fifty is always wrong or that parallelism is wasteful.

A model can choose a sensible amount of work. The runtime can enforce how much it will accept. Better delegation instructions help the model make that choice; the runtime still needs an enforced limit.

A semaphore can be the right answer

The strongest objection is correct. A single producer can acquire a permit before it creates the next task. With careful permit ownership and cancellation cleanup, that can bound unfinished child tasks without an explicit worker queue. The semaphore itself is not the mistake. Putting the wait inside thousands of already-created tasks is what gives those tasks somewhere to accumulate.

There are other perfectly reasonable choices. A backfill may intentionally accept a large backlog into durable storage, then feed compact job identifiers to workers. A small bounded batch may use gather because its maximum size is already known. Adding a scheduler to six independent operations could be needless ceremony.

I would ask a more specific question before changing any of them: when the downstream service stops making progress, which part of this system stops accepting new work?

If the answer is only "the model call," keep tracing upstream. Count callers waiting for a permit, callers waiting to enqueue, and jobs materialized before either wait. Then count retained results downstream. A pipeline that streams its input and keeps every output forever has another accumulation point.

Recursive delegation needs a separate check. If every worker holds its slot while waiting for child work that requires one of those same slots, a perfectly bounded system can deadlock. The flat worker example does not solve dependency scheduling. Limit the resource-consuming operation deliberately, and ensure waiting parents cannot occupy every slot their children need.

Put the limit where the work enters

For an agent runner, I want to see active work, queued work, and callers waiting for admission separately. I also want the age of the oldest waiting job. A queue of twenty jobs can represent a brief delay or a task the user abandoned an hour ago. The count alone cannot tell you which.

An item limit also leaves other resources unbounded. Ten short lookups and ten document-sized model requests consume different amounts of memory and tokens. Google's SRE chapter on overload makes the broader point that request counts can be poor proxies for resource cost. Apply the relevant budget where that resource is acquired; don't expect one integer to cover every constraint.

For interactive work, include admission wait in the deadline. When capacity runs out, decide whether the caller waits, gets an overload response, or accepts less work. If waiting is allowed, bound the callers allowed to wait too. Otherwise "we'll get to it" becomes an allocation policy that nobody wrote down.

The practical test is simple. Stall the workers. Keep presenting work. Observe what grows, and where the producer finally stops. If ten calls are active and everything else is accumulating off the dashboard, the concurrency limit is working. You still have a queue to design.