around LLMs. You’re working on a previously impossible challenge that is now possible to tackle using LLMs. You deploy the first few agents into your system, and everything seems fine. Customers are happy. The responses aren’t the fastest, LLMs tend to be quite slow, but it’s manageable.
As you start deploying more and more agents, however, you begin noticing strange anomalies. Slowly but steadily, latency starts increasing. At first, you assume it’s the LLM providers. Maybe the new OpenAI model is slower. Maybe Anthropic is under heavy load. LLMs are inherently slow anyway, right?
Then you start seeing timeout errors. When you dive into the logs and metrics, something doesn’t add up. OpenAI claims the requests are completing very quickly, yet your agent is taking an extremely long time to respond.
“OpenAI must be lying to us!”
Then comes the painful realization.
OpenAI wasn’t the problem. Our code was.
At Planck, we went through exactly that transition. As our agent ecosystem grew, so did our response times. This is the story of how we discovered the real bottleneck and how we solved it.
If you’re starting to use LLM agents in production and wondering how they will scale, if you’re an AI engineer, or if you simply enjoy challenging asynchronous system design problems, I invite you to follow along.
The Setup
At Planck, we had several LLM agents running in production. They were all served by a single service, meaning that a single request, whether via HTTP or a worker queue, would trigger all the agents simultaneously. Each agent would then make dozens of calls to its own sub-agents.
Here’s a simplified version of our code in Python:
```
import asyncio
import aiohttp
NUM_AGENTS = 5 # agents triggered by one request
CALLS_PER_AGENT = 30 # sub-agent LLM calls each agent makes
async def call_sub_agent(session: aiohttp.ClientSession) -> dict:
"""A single sub-agent step = one LLM call (io-bound)."""
payload = {
"model": "your-llm-model",
"messages": [{"role": "user", "content": "some fake prompt"}],
}
async with session.post("llm_url", json=payload) as resp:
return await resp.json()
async def run_agent(session: aiohttp.ClientSession, agent_id: int) -> list[dict]:
"""One agent fans out to dozens of sub-agent LLM calls at once."""
return await asyncio.gather(
*(call_sub_agent(session) for _ in range(CALLS_PER_AGENT))
)
async def handle_request() -> list[list[dict]]:
"""A single request triggers every agent simultaneously."""
async with aiohttp.ClientSession() as session:
return await asyncio.gather(
*(run_agent(session, agent_id) for agent_id in range(NUM_AGENTS))
)
```
At first glance, this architecture looks ideal. Almost all the work is I/O bound, so using Python’s async ecosystem allows every agent to execute concurrently. The latency of a request should therefore be dominated by the slowest LLM call, not by the total number of agents.
At least, that’s what we expected.
To test that, we used a local FastAPI server. Don’t worry, we verified that it responds quickly and is not the bottleneck.
We used a small randomasync sleeprange in the server endpoint to simulate the time it takes for an LLM to respond. We sent large payloads to the server and received large payloads back, which is quite common in LLM workflows.The results can vary depending on the payload size, the
async sleeprange, and the number of sub-agents per agent. So, don’t focus on the absolute numbers themselves, focus on the overall trend.
A graph showing response time as the number of agents increases (image by author)
Zooming In
Why could that be? We were using async exactly as we should. All of the agents and sub-agents were supposed to run simultaneously, so in theory there should have been little to no difference between running one agent and running 10,000 agents.
When we run the code in debug mode, we see event loop lag warnings, which means the task is finished, i.e the LLM returns a result, but the event loop can’t acquire a worker to look at it because it is busy doing something else, and in extreme cases this can lead to the timeout errors we were seeing in production.
WARNING:root: event-loop lag: 230 ms (loop was busy, not waiting)
WARNING:root: event-loop lag: 137 ms (loop was busy, not waiting)
WARNING:root: event-loop lag: 146 ms (loop was busy, not waiting)
WARNING:root: event-loop lag: 177 ms (loop was busy, not waiting)
WARNING:root: event-loop lag: 283 ms (loop was busy, not waiting)
...
But why can’t the event loop acquire a worker to handle the LLM response? In Python, because of the GIL, only one thread can execute code at a time. That means the event loop must be busy executing some CPU-bound work elsewhere.
But what could it be? To figure that out, we should use a profiler.
The Easy Wins
After reviewing the profiler, we can identify a few things. First, we waste a lot of time serializing and deserializing the body. By default, aiohttp uses the built-in json library, but we can significantly speed that up by using orjson.
Benchmark from the orjson documentation comparing deserialization and serialization performance
Second, we are not actually able to run all the I/O requests simultaneously. It turns out that the aiohttp session limits the number of HTTP connections we can open, and by default, that limit is 100.
So, we tried to increase and optimize this number, but not indefinitely. At some point, continuing to increase this value will hurt performance.
And if we rerun the tests, we see the results are much better.
However, even now, as the number of agents increases, the latency also increases.
A graph showing response time for different aiohttp connection limits (image by author)
The Real Bottleneck
Some of you might blame aiohttp, saying it is not fast enough or that, because it limits the number of connections, we still cannot truly parallelize I/O.
For that, and for similar lines of thinking, I want us to create an even simpler code example. Instead of relying on so many libraries and running real HTTP calls, let’s use async sleep to simulate I/O tasks and sleep to simulate CPU-bound tasks.
```
import asyncio
import random
import time
NUM_AGENTS = 50
CALLS_PER_AGENT = 30
IO_RANGE = (0.45, 0.55) # simulated LLM latency: I/O wait
CPU_RANGE = (0.001, 0.002) # simulated (de)serialization: CPU work
async def call_sub_agent() -> None:
await asyncio.sleep(random.uniform(IO_RANGE))
time.sleep(random.uniform(CPU_RANGE))
async def run_agent() -> None:
await asyncio.gather(*(call_sub_agent() for _ in range(CALLS_PER_AGENT)))
async def handle_request(num_agents: int) -> None:
await asyncio.gather(*(run_agent() for _ in range(num_agents)))
```
The result is quite similar to what we saw before. Of course, I cheated a bit and tuned the sleep times to create this beautiful graph, but even with different numbers, the important point remains the same: as the number of agents grows, latency increases. We cannot achieve true parallelism.
Of course, we can lower the curve by optimizing the CPU-bound tasks, but only to some degree.
A graph comparing the HTTP implementation with the simplified code (image by author)
At this point, it’s tempting to blame Python’s async runtime. Could it be that the event loop simply can’t handle this many concurrent tasks?
The event loop itself is incredibly efficient. If we benchmark an application that schedules hundreds of thousands of coroutines performing nothing but asyncio.sleep(), it scales remarkably well. As long as tasks are truly I/O-bound, asyncio has no trouble managing massive amounts of concurrency.
A graph showing bare asyncio event loop overhead as the number of tasks increases (image by author)
So why does our code behave so differently?
Because our tasks aren’t purely I/O-bound.
Every time an LLM response arrives, the event loop has to execute a small amount of CPU work before it can move on to the next coroutine, individually, these operations are almost free. Collectively, across hundreds or thousands of concurrent calls, they become the bottleneck.
Now some of you may start yelling: “Well, that’s what you get for using Python in production. Of course it won’t scale.”
But that misses the point. Sure, the GIL makes the issue worse, but even with a different language like Java or Go, there is still a limit to the number of processes you can open to parallelize CPU-bound tasks, because there is a limit to the number of cores your machine has.
Zooming Out
To find the right solution, we need to take a step back for a moment and stop focusing on which tasks consume the most CPU time and how we can optimize them. Instead, we need to recognize that there is a fundamental issue with our system’s design.
The important realization is that CPU work doesn’t scale the same way I/O does. Every completed I/O operation eventually requires some CPU work before the application can continue. Deserializing a response, counting tokens, validating data, or executing custom logic are all individually inexpensive operations. But when hundreds or thousands of agent calls finish around the same time, those tiny CPU tasks compete for the same CPU resources and eventually become the bottleneck.
Once we understood this, the solution became much clearer. Instead of trying to squeeze more work into a single event loop, we changed the architecture so that no single process was responsible for all agents.
Exactly how you do that depends on your architecture. You might use worker queues, multiple services, separate jobs, or another fan-out mechanism. The implementation details vary, but the underlying principle is always the same: distribute CPU work rather than asking a single process to do it all.
This did not eliminate the CPU work. Every response still had to be deserialized, validated, and processed. But instead of all that work accumulating in one process, it was distributed across multiple processes, CPU cores, and machines. That distinction is important. We did not eliminate the bottleneck, we divided it into smaller units that could be scaled horizontally.
In our case, we introduced a router that fanned out each request across multiple workers. Each worker was responsible for only a subset of the agents and had its own event loop and connection pool.
Distributing the workload solved several problems at once. It prevented latency from continuously increasing as we added more agents. It allowed teams to keep building new agents without worrying about slowing down existing ones. And it let engineers focus on solving business problems rather than spending time micro-optimizing every CPU-bound operation.
Final Thoughts
It’s easy to think of asynchronous code as infinitely scalable. After all, adding another coroutine feels almost free. But asynchronous I/O only removes waiting, it doesn’t eliminate computation. Eventually, every response still has to execute CPU tasks, and that’s where scalability begins to break down.
This isn’t really an AI-agent problem. It’s a fan-out problem. Whether a single request triggers hundreds of agents or a batch evaluation triggers thousands of LLM calls over a dataset, the underlying issue is the same: eventually, the CPU work required after each response becomes the bottleneck. Splitting the workload across multiple processes or jobs is often a much more scalable solution than trying to push everything through a single event loop.
The broader lesson extends far beyond Python or AI agents. Good engineering isn’t just about writing efficient code, it’s about designing systems that continue to scale as they grow. Well-written code cannot compensate for a poorly designed system. You can run profilers and optimize every line of code, but there is a limit to how much impact that can have.
Sometimes, you have to step back and ask yourself: does this approach really scale, or should we consider redesigning our system?
The story of how we actually split the workload is interesting in its own right. I’m thinking of making this into a series and writing more about that. Let me know if you’d be interested.
In fact, part of that redesign required moving our RAG pipeline from an in-memory store to a shared one. If you’re curious about that part of the story, you can read about it here:
https://towardsdatascience.com/when-not-to-use-vector-db/