Doing Several Things At Once

Agents spend almost all their time waiting. Waiting for a model, an API, a database. The work is not hard — it is slow, and it is slow somewhere else.

Concurrency is how you wait for several things at the same time. On a real agent it is often the difference between eight seconds and two.

This page assumes you have met async def and await in Async/Await Basics. Here we do the part that actually saves time.

The mistake everybody makes first

Writing async does not make anything faster.

async def slow():
    a = await fetch_a()    # 2 seconds
    b = await fetch_b()    # 2 seconds
    c = await fetch_c()    # 2 seconds

That takes six seconds.

await means "stop here until this finishes". Three stops, one after another. The word async did not make it fast — it made it possible to be fast.

gather runs them together

import asyncio

async def quick():
    a, b, c = await asyncio.gather(
        fetch_a(),
        fetch_b(),
        fetch_c(),
    )

Now it takes two seconds, not six.

gather starts all three, waits for all three, and hands back the results in the order you listed them — not the order they finished. So you can rely on position.

The bug hiding in the brackets

Look carefully at what is not there:

await asyncio.gather(fetch_a(), fetch_b())          # correct
await asyncio.gather(await fetch_a(), await fetch_b())   # wrong

In the wrong version, await fetch_a() finishes before fetch_b() has even started. You have carefully written gather and got sequential behaviour anyway.

You pass the coroutines to gather, and await gather itself.

See it for yourself

import asyncio, time

async def fetch(name, seconds=1):
    await asyncio.sleep(seconds)
    return f"{name} done"

async def main():
    start = time.time()
    a = await fetch("a"); b = await fetch("b"); c = await fetch("c")
    print(f"sequential: {time.time() - start:.1f}s")

    start = time.time()
    a, b, c = await asyncio.gather(fetch("a"), fetch("b"), fetch("c"))
    print(f"gathered  : {time.time() - start:.1f}s")

asyncio.run(main())

sequential: 3.0s
gathered  : 1.0s

Three seconds becomes one. That is the whole page in one output.

When it helps, and when it does not

Concurrency helps when the work is waiting on somebody else. It does nothing when the work is your own computer thinking.

Kind of work	Example	Does async help?
Network calls	model, API, database	yes, a lot
Reading many files	loading documents	yes
Maths, parsing, scoring on CPU	sorting, embeddings locally	no

For that last row you need multiple processes, not async. In agent work almost everything slow is the first two rows, which is why async is worth learning here.

Where it pays off in agent code

Several tool calls in one turn

A model asks for three tools at once. Running them one after another wastes the opportunity:

async def run_tools(calls):
    return await asyncio.gather(
        *[TOOLS[c["name"]](**c["args"]) for c in calls]
    )

The * spreads the list out into separate arguments for gather.

Fanning out over a list

Scoring fifty test cases, summarising thirty documents, one API call per member:

async def summarise_all(docs):
    return await asyncio.gather(*[summarise(d) for d in docs])

Careful with long lists — see the semaphore below.

When one of them fails

By default, if any task raises, gather raises immediately and you lose the successful results too. Rarely what you want.

results = await asyncio.gather(*tasks, return_exceptions=True)

for r in results:
    if isinstance(r, Exception):
        print("failed:", r)
    else:
        print("ok    :", r)

return_exceptions=True puts the exception into the results list instead of raising, so you decide what to do per item.

Nine good summaries and one failure? You almost always want the nine.

Do not launch a thousand at once

gather over a big list starts everything simultaneously. You will hit rate limits, exhaust connections, and the provider will start refusing you.

A semaphore is a doorman: only so many inside at a time.

sem = asyncio.Semaphore(5)

async def limited(doc):
    async with sem:
        return await summarise(doc)

results = await asyncio.gather(*[limited(d) for d in docs])

Five run at once, the rest queue. You keep most of the speed and stop being throttled. Five to ten is a sensible starting point for a paid API.

Timeouts

A hanging call should not hang your whole agent.

try:
    result = await asyncio.wait_for(slow_call(), timeout=30)
except asyncio.TimeoutError:
    result = "timed out"

Every external call in production wants a timeout. Without one, a single unresponsive service stops everything and the user sees nothing at all.

Running it

Inside an async def, use await. At the top of a script, start the loop once:

async def main():
    ...

if __name__ == "__main__":
    asyncio.run(main())

In a notebook there is already a loop running, so asyncio.run fails with "this event loop is already running". In a Colab cell just await directly:

result = await fetch("a")

Coming from Java or C#

C# will feel almost identical — Task.WhenAll is asyncio.gather. Java uses CompletableFuture.allOf, or virtual threads in recent versions.

Concept	Java	C#	Python
Async method	`CompletableFuture<T>`	`async Task<T>`	`async def`
Wait for one	`.join()`	`await`	`await`
Wait for many	`allOf(...)`	`Task.WhenAll`	`asyncio.gather`
Limit concurrency	thread pool size	`SemaphoreSlim`	`asyncio.Semaphore`
Timeout	`orTimeout`	`CancellationToken`	`asyncio.wait_for`

One important difference: Python's async is single-threaded. Nothing runs in parallel on separate cores — it interleaves waiting. That is why it helps with network calls and not with computation.

The upside: there are no locks to worry about between your own coroutines, which removes a whole category of bugs you may be used to.

Common mistakes

await in a loop when the calls are independent. The classic silent slowness.
await inside the gather brackets, which serialises the very thing you meant to parallelise.
Forgetting return_exceptions=True, so one failure discards nine good results.
No semaphore on a large fan-out, then blaming the provider for rate limiting.
Blocking calls inside async code. requests.get or time.sleep freeze everything — use httpx.AsyncClient and asyncio.sleep.
Expecting async to speed up computation. It will not.

Practise this

Run the sequential-versus-gathered example and see 3.0s become 1.0s.
Make one of the three fail. Watch gather lose everything, then add return_exceptions=True.
Fan out over twenty items with a semaphore of three and watch the pacing.
Wrap a slow call in wait_for with a one-second timeout and catch the TimeoutError.
Replace asyncio.sleep with time.sleep and confirm nothing overlaps any more.

Async Concurrency